.svg){kind=small}
Pushing live vehicle data to AWS S3
Amazon Simple Storage Service (AWS S3) provides an efficient solution for effectively organising, storing, and retrieving data. The organisation of data within AWS S3 is structured using two key components: buckets, which can be likened to folder structures, and objects, which represent individual files. As a deeply integrated service within the AWS ecosystem, AWS S3 is commonly utilised for various purposes, including the creation of data lakes, performing big data analytics, and supporting enterprise applications.
With the utilisation of the High Mobility platform, it becomes possible to seamlessly push live vehicle data directly to your AWS S3 bucket. The process does not require any additional hardware or coding, making it extremely convenient and user-friendly.
In order to start working with our AWS S3 integration, we are recommending going through each of the following steps. Every underlying aspect will be explained in more detail both in the video of our 10th Open Dev Talk and this blog post.
- Prepare your data container in the High Mobility console
- Set up up a new S3 bucket in AWS
- Link the data container and the bucket
- Test and validate the integration
Choosing data points for transmission
Before configuring AWS, it is essential to carefully choose the specific data points that you want to share from your fleet's cars. To begin the process, either create a new account on the High Mobility platform or log in to your existing account. Within the platform, you can create a new data container in Build mode. Another requirement is to add vehicles to your fleet. This step is necessary to activate them for the data transmission process. For further guidance, we have previously presented the best practices for creating data containers and provided detailed explanations of the fleet clearance API in our Open Dev Talks.
During the later stages of data transmission, each data point update will be represented as a data object. This object includes the Vehicle Identification Number (VIN), a unique message ID, a timestamp indicating the generation time of the data update, and the actual data value associated with the respective data point.
For more comprehensive information regarding the data structure and message payload, we encourage you to refer to our API references.
Configuring AWS S3
The entire process of configuring AWS and linking the High Mobility data container is explained in our publicly available documentation.
To begin, access your AWS account and navigate to the S3 service. Locate and click on the "Create bucket" button. Provide a globally unique name for the bucket as you enter the required information. Please note that High Mobility currently supports pushing data to the eu-west-1 (EU Ireland) region. Ensure that you have selected the correct AWS Region for your bucket.
By default, the option "Block all public access" is enabled. We strongly recommend keeping this option unchanged, as no public access is necessary for High Mobility to push data to the bucket.
Once the bucket is created, go to the "Permissions" tab. To grant access to High Mobility, we utilise the AWS schema for Bucket policy configuration. Click on "Edit" and copy and paste the supplied snippet from our documentation. The snippet should begin with the Version attribute.
Make sure to replace the three placeholders in the snippet as instructed. The "Principal" placeholder refers to the role of High Mobility, reflecting our service and granting the appropriate rights.
Following our documentation, we provide two configuration setups for both the sandbox and production environment. Depending on your scenario, paste the appropriate sandbox principal accordingly. The other two placeholders represent the bucket name. Replace these placeholders with the unique name of the bucket you created.
Finally, remember to save the changes after making the necessary adjustments.
Connecting High Mobility with AWS S3
To configure the AWS integration in our sandbox environment, open your data container and proceed to the "Data Streaming" section. Select the "AWS S3 Bucket" option and save the changes. A new popup will appear, prompting you to enter the bucket name in the respective field. Please provide the correct bucket name.
To validate the configuration you performed in the previous step, we recommend conducting a test transmission by clicking the "Test" button. During this test, a test file will be sent to the designated S3 bucket. High Mobility's platform will indicate whether the transmission was successful or not.
Returning to the S3 bucket itself, after refreshing the page, you will notice the creation of a new folder. The folder structure represents your app ID, which is the unique identifier of your data container. It is followed by the Auto API version level, year, month, date, and time. Upon navigating into the folder, you will find additional folders named "test" and "date." These folders hold JSON test files that can be downloaded to your local device if needed.
Testing the AWS S3 integration with simulated data
To fully test the S3 bucket integration, you can conveniently do so without the need to physically drive a car. In our sandbox environment, we provide free car simulators that generate data in the same format as a real car would. This allows you to simulate the car's behavior and collect relevant data for testing purposes. To learn more about our sandbox environment and its features, we encourage you to read further details provided in the respective article. If you haven't created a simulator yet, you can go back to the sandbox overview and navigate to the "Simulation studio" section. From there, you can create a new simulator associated with your fleet.
Returning to the "Develop mode," open your data container, and in the "Linked simulators" section, associate your simulator accordingly. Once the simulator is opened, you can manipulate the data points listed in the "Capabilities" view. For instance, you can set the odometer to new values and adjust the fuel level accordingly.
Our sandbox environment automatically pushes these data changes to the designated S3 bucket. The console in the sandbox displays a list of all the updates pushed, ensuring transparency and visibility into the transmitted data. It's important to note that there might be a delay between the actual data transmission and the data being received and persisted in the S3 bucket. Data is batched into separate JSON files every five minutes or when 10,000 messages have been received.
If you require more real-time access to connected car data, you may consider switching to our MQTT stream, which offers faster data updates.
-------------------------------
High Mobility Open Dev Talks
At High Mobility, we are passionate about new technology. We offer free open source tools and developer friendly documentation for any projects to be integrated smoothly. More than 800 developers and product managers have already signed up for our moderated community platform and we are hosting connected car competitions for your innovative, connected car ideas.
Join our community on Slack
