Redshift Spectrum
A Redshift feature that allows users to run queries against data stored on Amazon S3.
Geospatial chart
A QuickSight visual type that is best suited for displaying different data values across a geographical map.
Amazon Kinesis
Data Streams that can collect and process large streams of data records in real time.
Amazon Macie
A fully managed data security and data privacy service that uses machine learning and pattern matching to discover and protect your sensitive data in AWS.
Amazon Athena
An interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.
True
True or False. A single COPY command is faster than multiple COPY commands when loading one Redshift table from multiple files.
AWS Glue
A serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development.
Not studied (3)You haven’t studied these terms yet!
Per-query limit
A form of cost control in Amazon Athena that limits the amount of data scanned per query.
Amazon S3
An AWS service that is used as a referenced data source for a Kinesis Data Analytics application?
Amazon S3 Glacier
A secure, durable, and extremely low-cost Amazon S3 storage class for data archiving and long-term backup.
Types of Data Collection
Real Time
Near Real Time
Batch
Real Time Data Collection Services
Kinesis Data Streams
Simple Queue Service
Internet of Things
Near Real Time Collection Services
Kinesis Data Firehose
Database Migration Service
Batch Collection Services
Snowball
Data Pipeline
Kineses Services
Kinesis Streams
Kinesis Analytics
Kinesis Firehose
Kinesis Streams
Steams are divided in ordered Shards
Data retention is 24 hours by default, 7 day max
Ability to reprocess data within the data retention period
Once data is inserted, it cannot be deleted
Kinesis Streams Records
Data Blob
Record Key
Sequence Number
Data Blob
Data being sent, serialized as bytes. Up to 1MB. Can represent anything
Record Key
Sent alongside a record, helps to group records in Shards. Same key = Same shard
Use a highly distributed key to avoid the “hot partition” problem
Sequence Number
Unique identifier for each records put in shards. Added by Kinesis after ingestion
Kinesis Streams Limits (Producer)
1MB/s or 1000 messages/s at write per shard.
Kinesis Producer Components
Kinesis SDK
Kinesis Producer Library (KPL)
Kinesis Agent
3rd Party Libraries
Kinesis Streams Limits (Consumer Classic)
2MB/s or 5 API calls/s per shard across all consumers
Kinesis Streams Limits (Consumer Enhanced)
2MB/s per shard, per enhanced consumer. No API calls needed
Kinesis Producer SDK (PutRecord(s))
API’s that are used are PutRecord (one) and PutRecords (many records).
PutRecords uses batching and increases throughput (less HTTP requests)
Kinesis Producer SDK (Use Cases)
Low throughput
Higher latency
Simple API
AWS Lambda
Kinesis Data Streams (Manage AWS Services)
CloudWatch Logs
AWS IoT
Kinesis Data Analytics
ProvisionThroughputExceeded Exceptions
Problem:
Happens when sending more data (exceeding MB/s or TPS for any shard)
Make sure you don’t have a hot shard (such as your partition key is bad and too much data goes to that partition
Solution:
Retries with backoff
Increase shards (scaling)
Ensure partition key is distributed
Kinesis Producer Library (KPL)
Easy to use highly configurable C++/Java library
Used for building high performance, long-running producers
Automated and configurable retry mechanism
Synchronous or Asynchronous API
Submits metrics to CloudWatch for monitoring
Batching - increase throughput, decrease cost
Compression must be implemented by user
KPL Records must be de-coded with KCL or special helper library
Kinesis Agent
Monitor Log files and sends them to Kinesis Data Streams
Java-based agent, built on top of KPL
Install Linux-based server environments
Write from multiple directories and write to multiple streams
Routing based on directory/ log file
Pre-process data before sending to streams
The agent handles file rotation, checkpointing, and retry up failures
Emits metrics to CloudWatch for monitoring
Kinesis Consumers Classic Components
Kinesis SDK
Kinesis Client Library
Kinesis Connector Library
3rd party libraries
Kinesis Firehose
AWS Lambda
Kinesis Consumer Classic SDK (GetRecords)
Records are polled by consumers from a shard
Each shard has 2MB total aggregate throughput
GetRecords returns up to 10MB/s of data (then throttle for 5 seconds) or up to 10K records/s
Maximum of 5 GetRecords API calls per shard per second = 200ms latency
If 5 consumer applications consume from the same shard, means every consumer can poll once a second and receive less than 400KB/s
VirtuaBox adalah tools / perangkat virtualisasi untuk menginstall sistem operasi didalamnya biasanya virtualbox digunakan sebagai alat / tools pengujian, membantu lingkungan development.
Apa Itu Virtualbox Headless Vs GUI ?
VirtuaBox Headless adalah virtualbox yang menjalankan virtual machine secara background tanpa adanya interkasi GUI sedangkan GUI sebaliknya ada interaksi GUI.
Apa Itu PHPVirtualBox ?
PHPVirtuaBox adalah tools / perangkat yang mendukung virtualbox client web dalam manajemen virtualisasi.
Prasarat yang dibutuhkan untuk menjalankan VirtualBox dan PHPVirtualBox adalah sebagai berikut.
OS & Tools (Prerequisite):
VirtualBox 7.0.2
PHPVirtualBox (Hanya Mendukung VirtualBox 6.1.0)
OS Ubuntu 18.04 Bionic
Install VirtualBox 7.0.2 + PHPVirtualBox di Ubuntu 18.04 Bionic
Running VirtualBox Headless Via CLI
Start VirtualBox VM in Headless Mode
List virtual machines
VBoxManage list vms
“MyVM” {e4b0c92c-4301-4a7d-8af8-fe02fed00451}
This version of phpVirtualBox (6.1-0) is incompatible with VirtualBox 7.0.2. You probably need to download the latest phpVirtualBox 7.0-x. See the Versioning section below the file list in the link for more information
Notice diatas dikarenakan phpvirtualbox 6.1 tidak mendukung dengan virtualbox 7.0.2 karena phpvirtualbox pengembangannya stuck alias kurang jadi solusinya pakai versi develop.
Untuk Solusi :
Pakai PHPVirtualBox dengan branch develop / master saya biasanya pakai develop karena saat direview cukup stable dan Downgrade VirtualBox dengan Versi 6.1.0 karena phpvirtualbox hanya mendukung VirtualBox sampai dengan versi 6.1.0
karena lambat pengembangannya phpvirtualbox belum lagi karena segudang sekuritas dan bug nya jadi sebenarnya virtualbox ini punya gandengan yang lebih hebat secara stack management virtualisasinya ya kita kenal dengan Vagrant by HashiCorp, semua environment bisa kita racik sesuai kebutuhan.
How VM Machine Enable Autostart when booting?
Auto Start Virtual Box VM’s on Linux
Create Config Files
/etc/default/virtualbox
sudo nano /etc/default/virtualbox then add:
VBOXAUTOSTART_DB=/etc/vbox
VBOXAUTOSTART_CONFIG=/etc/vbox/autostart.cfg
/etc/vbox/autostart.cfg
sudo nano /etc/vbox/autostart.cfg then add:
Then, for each allowed username: sudo usermod -aG vboxusers USERNAME
:~$ ls -l /etc/vbox
total 8
-rw——- 1 USERNAME USERNAME 1 Dec 14 01:37 USERNAME.start
-rw-r–r– 1 root root 179 Dec 14 01:31 vbox.cfg
Note: Log out and in for it to apply.
VBoxManage: error: Adding machine ‘Ubuntu-VM’ to the autostart database failed with VERR_ACCESS_DENIED If you this error message, manually create .start file under /etc/vbox
Choose VMs to automatically start
Run the on-time command VBoxManage setproperty autostartdbpath /etc/vbox.
Add VM’s!
VBoxManage list vms
VBoxManage modifyvm <uuid|vmname> –autostart-enabled <on|off>
Restart VirtualBox autostart service
sudo service vboxautostart-service restart or reboot the machine.
Altenrative 1 :
This is working ok with Ubuntu server 12.04 and VirtualBox 4.2.20.
1) Create the startup script file
in /etc/init.d - sudo nano /etc/init.d/StartVM.
#! /bin/sh
/etc/init.d/StartVM
#
#Edit these variables!
VMUSER=username
case “$1” in
start)
echo “Starting VirtualBox VM SMARTHOST …”
sudo -u $VMUSER VBoxManage startvm SMARTHOST –type headless
echo “Starting VirtualBox VM wxp-acceso …”
sudo -u $VMUSER VBoxManage startvm wxp-acceso –type headless
echo “Starting VirtualBox VM wmmaq_edi …”
sudo -u $VMUSER VBoxManage startvm vmmaq_edi –type headless
;;
stop)
echo “Saving state of Virtualbox VM SMARTHOST …”
sudo -u $VMUSER VBoxManage controlvm SMARTHOST savestate
echo “Saving state of Virtualbox VM wxp-acceso …”
sudo -u $VMUSER VBoxManage controlvm wxp-acceso savestate
echo “Saving state of Virtualbox VM vmmaq_edi …”
sudo -u $VMUSER VBoxManage controlvm vmmaq_edi savestate
;;
*)
echo “Usage: /etc/init.d/StartVM {start|stop}”
exit 1
;;
esac
exit 0
And
2) Give the script executable permission
sudo chmod +x /etc/init.d/StartVM
ls -l /etc/init.d/StartVM
-rwxr-xr-x 1 YOUR_USERNAME YOUR_USERNAME 468 Nov 23 12:39 /etc/init.d/StartVM
and
3) Tell script to run at startup.
sudo update-rc.d StartVM defaults 99 01
sudo update-rc.d StartVBox defaults
Notes :
Warning vboxdrv kernel module is not loaded
add line on /etc/init.d/StartVM :
sudo modprobe vboxdrv
Virtual Box Settings to automatically run VMs on boot
I had this problem too. The issue is that the autostart service script is trying to include /etc/vbox/vbox.cfg just like it does with /etc/default/virtualbox. You need to use a different filename for the autostart config.
Autostart Virtualbox VMs
So… certainly things evolve quickly and others lack behind. We’ve got systemd since 15.10 or so and it was the prominent change in 16.04. We actually made the switch for Snap! Websites and that help tremendously as it is very reliable since then (older versions were really bad in comparison.)
At first, I found a Stackoverflow answer describing the old way and it did not work which is why I decided to switch to systemd. I found this page about it and it’s already pretty good! Just a couple of things I had to tweak on my system to make it all work properly.
Click to learn more on how to manage an environment of VirtualBox computers
Click to learn it all
(available on Amazon)
Deleting the Old Way
Just to make sure it works properly, delete the files from the old environment if you created them. If you did edits to some of them, you probably want to undo those edits.
This is to make sure that the old way doesn’t suddenly kick in and mess up your systemd setup.
$ rm -rf /etc/vbox
On my end, I edited the /etc/default/virtualbox file to add these two variables:
VBOXAUTOSTART_DB=/etc/vbox
VBOXAUTOSTART_CONFIG=/etc/vbox/autostart.cfg
So I went ahead and deleted those two lines.
However, I also changed the following and that I kept:
#SHUTDOWN=poweroff
SHUTDOWN=acpibutton
I prefer to have a good APCI shutdown and a plain power off. If it takes too long to shutdown, we’ll anyway get a power off (a.k.a. a kill).
If you did anything else to try to make the old way work, please make sure to revert that too. You’re on your own on that side of things…
Create Service File
Now you are ready to create a service file.
In mine, I have pretty much the same thing as the one from the page I referenced above, except for the vboxdrv.service. I’m not too sure where that came from, but under Ubuntu 18.04, there is no such thing. Though I remove that one reference.
The script includes two names you need to change:
vm1 — enter the name of your Virtualbox machine (this one appears three times)
alexis — unless you are also an alexis and your account is name like that, change that name
Then save the file under /etc/systemd/system/.
For example, if you use gvim to edit your files:
$ sudo gvim /etc/systemd/system/vm1.service
Then copy/paste the following and do the edits as expected:
[Install]
WantedBy=multi-user.target
Note: the vboxdrv.service comes from a comment (see below) and should help even further (i.e. I have had problems with my services starting once in a while, most of the time, I still had to restart them by hand—I’ll have to confirm once I reboot that it works every time).
Once the file is ready, you need to tell systemd to reload everything including that new file. It may do so automatically, but just in case here is the command you need:
sudo systemctl daemon-reload
Now you are ready to use this service.
Note: To see a list of services you have running on your system, the cleanest way is to use the following command:
sudo systemctl
It automatically sends its output to less so you don’t have to pipe manually. At the bottom of the list comes a brief explaination of what the columns stand for. Especially, if a column says “failed”, then the service doesn’t work as expected.
Enable/Start/Stop Service
Now that we created a service file, we want to enable and start it. This supposes that your VM is ready to be started as if you were rebooting your machine.
Note that in the script above, there is nothing to say that one service should be started before and/or after another. It’s doable but you should ask yourself whether it makes sense that the computers should be started in a very specific order. i.e. if something fails on computer B when A isn’t running, it should keep trying until it works…
First, you can test that systemd recognize your file with:
$ systemctl status vm1
The status command is also useful to check whether a service is running or not.
Now you can start the VM.
$ sudo systemctl start vm1
It should return on the very next line unless something goes wrong.
If you have the Virtualbox UI open, you should see that it gets started there. The icon changes with what appears on the screen as normal. However, because we use the headless feature, you don’t see the actual GUI part. You can show that window at any time while the machine is running by selecting the “→ Show” menu (it’s found in the Machine menu or when you right-click on the VM).
Finally, to make the VM run automatically on a reboot, you need to enable it. This is done using the enable command like so:
$ sudo systemctl enable vm1
You can verify the current status at any time. Note that the enable state shows on its own. There are two entries in the output. Both should be enabled. Here is an example of the output. I highlighted the enabled you are looking for in red:
$ systemctl status vm1
● virtualbox_finball2.service - finball2
Loaded: loaded (/etc/systemd/system/vm1.service; enabled; vendor preset: enabled)
Active: active (exited) since Sat 2019-11-23 20:13:36 PST; 34s ago
Tasks: 0 (limit: 9830)
CGroup: /system.slice/vm1.service
Nov 23 20:13:35 monster systemd[1]: Starting vm1…
Nov 23 20:13:36 monster VBoxManage[12050]: Waiting for VM “vm1” to power on…
Nov 23 20:13:36 monster VBoxManage[12050]: VM “vm1” has been successfully started.
Nov 23 20:13:36 monster systemd[1]: Started vm1.
The last few lines are logs (called journals in systemd) showing what happened. It should say that it started your Virtualbox system.
You can later disable the service like so:
$ sudo systemctl disable vm1
which will prevent it from auto-restarting on a reboot. Note that you can still start a VM marked as being disabled.
Renaming a VM
I use rather long names for my VM that include:
The name of what I’m using the VM for
The OS name (Ubuntu, Fedora, FreeBSD, SunOS, MacOS, etc.)
The hostname of the computer running the VM (because when you run VMs on 3 or 4 different computers, it’s not easy to know what’s what otherwise)
The processor (i86x or amd64)
The sub-name of the computer (i.e. I create clusters that get one VM name + one sub-name such as front end, database, backend…)
I think such long names are great in the Virtualbox interface, but to type in your service file, it may cause problems (between dashes, spaces, parenthesis… and just the fact that such long names are hard to replicate without a few mistakes into them…)
So on my end, I had to rename the VMs. The cool thing, though, is that we now have grouping capabilities. So I can still keep my crazy naming convention for the group, and name each computer in the group something really simple (i.e. _ — for example wordpress_frontend and wordpress_database).
Where is My VM’s Window?
The script is going to start the VM headless. This is actually important if you were to run the VM on a server without X-Windows. (Yes! It’s doable, I’m just not too sure how you do the installation all by hand. It must be somewhat painful!)
Your VMs are going to start alongside the X-Windows environment and because of that, the X-Windows system may not be ready to open a window in the first place. So instead we start VMs headless.
When you are in the Virtualbox GUI, you see the VMs that are running. They have a green arrow. Right-click on those and select “→ Show” to open the window. Now you have full access to the VM’s console. To hide the window again, you can of course minimize. However, if you want to actually close the X-Windows but leave the VM running, use the “Detach GUI” option found under the “Machine” menu.
In older versions (before 5.x), detaching the GUI was done in various ways:
Machine » Detach GUI
View » Close VM
Close VM (as a button)
Various Errors
The virtual machine has terminated unexpectedly during startup with exit code 0 (0x0).
Looking into that one, I could not really see why one of my virtual machine wouldn’t start, especially with an exit code of 0… First I checked that the machine still existed as expected (right location, etc.) It was there.
So next I searched the Internet and found a page which I won’t even link here, it was totally useless except for one small detail: one of the message was one person asking the other what they saw in their log file. Yes. Why wouldn’t I look at that for once.
Here are the last few messages:
00:00:01.005004 GIM: KVM: Resetting MSRs
00:00:01.186998 VMSetError: /build/virtualbox-p8NxA3/virtualbox-5.2.34-dfsg/src/VBox/VMM/VMMR3/VM.cpp(326) int VMR3Create(uint32_t, PCVMM2USERMETHODS, PFNVMATERROR, void, PFNCFGMCONSTRUCTOR, void, VM, UVM); rc=VERR_VMM_R0_VERSION_MISMATCH
00:00:01.187018 VMSetError: The VMMR0.r0 module version does not match VBoxVMM.dll/so/dylib. If you just upgraded VirtualBox, please terminate all VMs and make sure that neither VBoxNetDHCP nor VBoxNetNAT is running. Then try again. If this error persists, try re-installing VirtualBox.
00:00:01.188756 ERROR [COM]: aRC=NS_ERROR_FAILURE (0x80004005) aIID={872da645-4a9b-1727-bee2-5585105b9eed} aComponent={ConsoleWrap} aText={The VMMR0.r0 module version does not match VBoxVMM.dll/so/dylib. If you just upgraded VirtualBox, please terminate all VMs and make sure that neither VBoxNetDHCP nor VBoxNetNAT is running. Then try again. If this error persists, try re-installing VirtualBox. (VERR_VMM_R0_VERSION_MISMATCH)}, preserve=false aResultDetail=0
00:00:01.189069 Console: Machine state changed to ‘PoweredOff’
00:00:01.194933 Power up failed (vrc=VERR_VMM_R0_VERSION_MISMATCH, rc=NS_ERROR_FAILURE (0X80004005))
00:00:01.695786 GUI: UIMachineViewNormal::resendSizeHint: Restoring guest size-hint for screen 0 to 800x600
00:00:01.695873 ERROR [COM]: aRC=E_ACCESSDENIED (0x80070005) aIID={76eed314-3c72-4bbb-95cf-5eb4947a4041} aComponent={DisplayWrap} aText={The console is not powered up}, preserve=false aResultDetail=0
00:00:01.695961 GUI: Aborting startup due to power up progress issue detected…
And interestingly enough, they give you the exact answer of what to do next (3rd line in the logs above):
Sstop all your VMs;
Quit VirtualBox;
Make sure all the VM services are down (reboot if you want to make 100% sure); and
Restart VirtualBox.
Now it should work and it did for me.
The Old Way
At first, I tried the old way, mainly because there are a large number of posts in link with that one. But I just could not find the Virtualbox autostart script anywhere. I tried that once and nothing happened so I would imagine that it’s missing (was removed) in new versions (Ubuntu 18.04 at least).
These are a few points I wanted to make regarding that method.
I go the following error when trying to install a Virtualbox in auto-start mode and trying to fix the problem took me a little while…
VBoxManage: error: Adding machine ‘my-virtual-box’ to the autostart database failed with VERR_ACCESS_DENIED
VBoxManage: error: Details: code NS_ERROR_UNEXPECTED (0x8000ffff), component SessionMachine, interface IMachine, callee nsISupports
VBoxManage: error: Context: “COMSETTER(AutostartEnabled)(ValueUnion.f)” at line 3035 of file VBoxManageModifyVM.cpp
The fact is that once you added yourselves to the vboxusers group, you need to log out and back in properly or even just forget about that group for a moment and use your primary group instead.
In other words, the docs I’ve found such as this one:
tell you to use vboxusers but you could also do that:
chgrp /etc/vbox
(Note: in most cases, your primary group name is your username)
and the chmod is not even necessary. Of course, that one won’t help much if you want many users to be able to use the autostart feature. The idea is just to have a way to do your setup before you log out and/or test with a full restart of your computer (in my case, I have a monster with 512Gb of RAM, two processors with a total of 64 threads, it takes forever to reboot so I prefer to avoid it if I can.)
I’ll try again and see whether it works if I restart the computer and I have the correct group in there…
However, I just rebooted and it did not work, there is still something fishy about it under 18.04, I just can’t see the vbox service. I can see the Virtualbox kernel module, just no the service. I probably need to install something else…
Add new comment
Re: Autostart Virtualbox VMs in Ubuntu 18.04
Submitted by Anonymous (not verified) on Thu, 10/07/2021 - 18:33.
I’m running mint linux.
Noticed I was getting an error:
Oct 07 20:52:38 ShedPC VBoxManage[4984]: WARNING: The vboxdrv kernel module is not loaded. Either there is no module
Oct 07 20:52:38 ShedPC VBoxManage[4984]: available for the current kernel (5.4.0-88-generic) or it failed >
Oct 07 20:52:38 ShedPC VBoxManage[4984]: load. Please recompile the kernel module and install it by
Oct 07 20:52:38 ShedPC VBoxManage[4984]: sudo /sbin/vboxconfig
Oct 07 20:52:38 ShedPC VBoxManage[4984]: You will not be able to start VMs until this problem is fixed.
And then later on down the log, vboxdrv would load
Had to add the following lines to my config:
[Unit]
Description=FlexRadio
After=network.target virtualbox.service vboxdrv.service
Requires=virtualbox.service vboxdrv.service
Before=runlevel2.target shutdown.target
Don’t know if I actually needed the Requires line, but seemed like a thing to do…
Many thanks for the webpage entry. It was of great help, easy to follow and well written.
reply
Re: Autostart Virtualbox VMs in Ubuntu 18.04
Submitted by Alexis Wilke on Thu, 01/02/2020 - 20:29.
I’m glad that my post helped you find out how to make it work on your machine.
On my end, if I run:
locate virtualbox.service
nothing is output. So I’m sure I do not have such a file and of course I can’t wait on it being up/run if I want my boxes to start automatically. That being said, for me it works without that additional name.
That being said, there’s a virtualbox script under /etc/init.d and it may be that your script will wait for that one to have run. That script makes sure that all the drivers are loaded before trying to start virtual boxes that are marked for auto-headless startup. But that did not work for me.
Update: The fact is that now I see the virtualbox.service file! I’m not too sure how/what changed, maybe I had to reboot or something. The fact is the last time I rebooted, my VMs did not start and since then I see that the virtualbox.service exists so I can depend on it. I updated my post accordingly.
reply
Re: Autostart Virtualbox VMs in Ubuntu 18.04
Submitted by Anonymous (not verified) on Thu, 01/02/2020 - 07:27.
Hi there, I followed your instructions, but found that adding network.target on the After line in /etc/systemd/system/vm1.service as you have suggested here was insufficient.
I found that without also adding an additional virtualbox.service entry on the After line my VMs were unable to find character device /dev/vboxdrv at boot and would fail to start.
As you have rightly point out in your article vboxdrv.service does not appear to exist on Ubuntu 18.04. However please be aware that Ubuntu DOES have the similarly named virtualbox.service available instead and we must wait for this to launch before loading any VMs.
In conclusion, in my own /etc/systemd/system/vm1.service file I have the following two lines which seems to work fine for me in Ubuntu 18.04:
VBoxManage startvm win_7_ultimate –type headless
WARNING: The vboxdrv kernel module is not loaded. Either there is no module available for the current kernel (5.4.0-132-generic) or it failed to load. Please recompile the kernel module and install it by