| Making Vybrid Visible as USB Storage Using Linux |
 |
|
The most typical example of when this functionality is required is a "data harvesting" application. In such an application, a remote Vybrid device collects data readings from various sensors and then stores collected readings into a non-volatile storage, such as NAND Flash. Once in a while, a technician visits the remote site to offload harvested data and take it to a main site for further processing. Using the functionality described in this note, the data offloading is as simple as connecting the Vybrid as a USB device to a technician's notebook, which immediately recognizes it as USB storage. Collected data can then be simply copied from the USB storage to other disks on the notebook. A variation of the same application is a movable device that is periodically taken to a main office for data offloading. In this scenario, the embedded device is simply connected to a PC as a USB device and data is copied from the USB storage to the host computer. Hardware Platform The hardware platform is theEmcraft Systems Vybrid VF6 system-on-module (SOM) plugged into the TWR-VF6-SOM-BSB development baseboard. High-Level Architecture We will use the Linux File-backed Storage Gadget (FSG) to implement what we need. The main idea is that the FSG presents a file in the local file system as a storage device to the USB host. On the target side, the content of the disk is stored in a file (hence the "File-backed Storage"). The backing file is a "Unix file" that is given to the File-backed Storage Gadget as a parameter, which can be a normal file or a block device, such as a disk partition. To make data sharing workable using a Windows host, the USB storage must be formatted as a FAT file system. This way, data gathered by the target into the USB storage can be copied or manipulated otherwise, using standard Windows tools on the USB host end. The implication of the above is that the backing file must be mounted as a FAT32 file system on the target side. This is not really a problem since Linux supports the FAT32 file system (along with many other file systems). Clearly, a file system can be mounted on a block device such as a disk or a disk partition but it is also possible to mount a file system on a normal file using the so-called "loop" device. The big question is where we want the backing file to reside on a Vybrid-based device. We need the file to be on a non-volatile media. It is not really a hard requirement and we could keep the backing file in a RAM-based filesystem too, however in case there is an unexpected power cut, which happens all the time in embedded applications, all data we have collected would be lost. Not really acceptable, in a generic application. The backing file must be in Flash, then. The Vybrid VF6 system-on-module (SOM) provides a 128 MBytes SPI Flash. In Linux, that Flash can be split on as many partitions as makes sense from the application point of view. A few first MBytes of the SPI Flash are taken by the U-boot environment variables and a bootable Linux image, however the remaining Flash is sufficient to allocate a partition specifically for hosting the USB storage backing file. Now, having allocated a Flash partition to the USB backing file, we have two options, potentially:
To summarize it all, the architecture we want to implement will be as follows:
Building the Demo In order to test the USB gadget functionality, apply this patch to the Emcraft release 1.14.0 kernel tree. Then, make the following changes to the networking project included in the Emcraft software distribution. On the development host, go to projects/networking and run the make kmenuconfig command to access kernel configuration menu. Navigate to the Device Drivers -> Block Devices menu and enable the Loopback device support menu item:
Navigate to the Device Drivers -> USB Support menu and enable the USB Gadget Support item:
Then enter the USB Gadget Support menu and enable the File-backed Storage Gadget kernel module:
Run the make bmenuconfig command to enter the Busybox configuration menu. Navigate to the Coreutils menu and enable the dd menu item:
Navigate to the Linux System Utilities menu and enable the losetup menu item:
In the target file system, there must be appropriate device nodes to access the loopback device using standard Linux tools and utilities. Add the following lines to your <project>.initramfs file to create the device nodes for the loopback device: nod /dev/loop-control 0600 0 0 c 10 237 Also, the losetup utility and the kernel module must be added to the target file system. Add the following lines to your <project>.initramfs file: slink /bin/losetup busybox 777 0 0 Preparing USB Storage This section explains how to prepare the USB storage for deployment. This command sequence is a once-off procedure that needs to be performed on the Vybrid at software manufacturing time. With the bootable Linux image (from the previous section) installed, U-Boot loads the Linux image from the NAND Flash to the RAM and passes control to the kernel entry point: U-Boot 2011.12-vf6-1.14.0 (May 14 2015 - 19:50:44) The kernel proceeds to boot-up, initializing the configured I/O interfaces and sub-systems: Starting kernel ... First thing we need to do is to erase the Flash partition we are going to use for the JFFS2 file system: ~ # flash_eraseall -j /dev/mtd5 We can now mount the newly created JFFS2 file system. As expected, it is empty at this time: ~ # mount -t jffs2 /dev/mtdblock5 /mnt Next step is to create in the JFFS2 filesystem a backing file for the FAT32 image. The JFFS2 partition is about 80 MBytes in size so let's create a 32 MBytes FAT32 file system image: ~ # dd if=/dev/zero of=/mnt/fat32.part bs=1M count=32 Now, let start the File-backed Storage Gadget on the backing file: ~ # insmod /g_file_storage.ko file=/mnt/fat32.part stall=0 removable=1 Assuming the Vybrid is connected to a host as a USB gadget, the above command will make it immediately visible as a USB storage device to the host. Or, the Vybrid is not connected to a host at the time the above command is run, simply connect it to a host using an appropriate USB cable. Specifically, connect a USB 2.0 A Male to Mini-B cable between a free USB port on the host and the P13 USB interface connector on the TWR-VF6-SOM-BSB-2A baseboard. At this point the Vybrid should become visible as a USB storage device to the host, however the storage still needs to be formatted as a FAT32 file system to allow using it with standard host software. The formatting is done on the host using standard software. For instance, on a Windows machine, go to My Computer and then locate an icon for the new removable disk. Ask to format it and agree with anything that Windows suggests, until the storage is formatted as a FAT32 file system. Harvesting Data on the Target Ok, so now everything is ready to start deploying the Vybrid as a "data harvesting" device. In all probability, the commands shown below would be put into the /etc/rc start-up script by any reasonably application, but in order to make the set-up command sequence easier for understanding, let's run the required commands manually. For the sake of running a clean test, let's power-cycle the device. As expected, Linux comes up to the shell: Freeing init memory: 3452K First thing is to mount the JFFS2 file system: ~ # mount -t jffs2 /dev/mtdblock5 /mnt Next step is to associate the backing file with a loopback block device, which allows us to mount a FAT32 file system on the backing file: ~ # losetup /dev/loop0 /mnt/fat32.part Now, it is time to start the File-backed Storage Gadget making the FAT32 file system mounted on the backing file visible to a USB host as a storage device: ~ # insmod /g_file_storage.ko file=/mnt/fat32.part stall=0 removable=1 Let's "harvest" some data and store what is collected into a file in the FAT32 file system. In this demo, we emulate a data stream by taking a snapshot of the system time each second: ~ # while date >> /tmp/m/data.log; do sleep 1; done Having let the "data harvesting" run for a while, let's interrupt it (by pressing ^-C) and take a look at what data we have collected: ^C All looks as expected, so we are ready to unmount the device and try accessing the FAT32 file system from a USB host. ~ # umount /dev/loop0 Processing Data on the Host Given the above set-up, a host will be able to see the Vybrid as a removable disk as soon as a USB connection is made between a free port on the host and the USB P13 connector on the target. A message like this will appear on the target console when a connection to the host has been made: ~ # g_file_storage gadget: high speed config #1 Host software should recognize the newly plugged USB device as a storage device and mount it as a FAT32 file system. Data collected on the target can be copied to other host disks for further processing. Note that the format of Windows and Unix text files differs slightly. In Windows, lines end with both the line feed and carriage return ASCII characters, but Unix uses only a line feed. As a consequence, some Windows applications will not show the line breaks in Unix-format files. Assuming that data is stored in a text file (vs a binary file) and Windows is a data processing host, Linux data harvesting applications should take care of the difference by adding a carriage return character to data logs. Data Synchronization Issues It is important to note that it is not safe to update the USB storage both on the target and host sides at the same time. Here is what the File-backed Storage Gadget documentation has to say about this: AN IMPORTANT WARNING! While FSG is running and the gadget is connected to a USB host, that USB host will use the backing storage as a private disk drive. It will not expect to see any changes in the backing storage other than the ones it makes. Extraneous changes are liable to corrupt the filesystem and may even crash the host. Only one system (normally, the USB host) may write to the backing storage, and if one system is writing that data, no other should be reading it. The only safe way to share the backing storage between the host and the gadget's operating system at the same time is to make it read-only on both sides. Given that the primary use of this technology is to let the host copy harvested data for further processing, this should be an acceptable limitation for most applications. It is important to understand however that the host will see the USB storage in a state it was in when the USB cable was connected between the target and the host. Further updates made to the storage by the target will not be visible on the host until a next plug-in. As mentioned above, updates made on the host side may be unsafe in case the target continues harvesting data when the host is connected. Probably, the best strategy is to assume a read-only mount of the USB storage on the host side and let application code on the target take care of purging data at appropriate times. Note the File-backed Storage Gadget provides a special flag (ro=1) to allows mounting the USB storage for read-only access by the host. |
