FAT SD-Card is nice to use, but (very) slow

For questions and comments on programming in general. And for any items that don't fit into the forums below.

Moderators: Benj, Mods

Post Reply
stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Hi Ben
At first, the Fat from FC is realy nice to use and is working fine. It's create file by file on a card.
I have PIC 18LF6722. It's working on 32Mhz. There is a SD Card on the board. The card is working with clock/4
I used the Blockwrite (write file sector). The goal is to sample 2Channel 16Bit with 2kHz. For this I used two buffers.
If I sample data in buffer_A and I write buffer_B to the disk. If the buffer is full I change the buffers.
That's working fine. To write 512Byte (one Block) to of the SD-card, needs 25mSek. It's ok....For the "move to the next sector" needs 5mSek. It's ok. But every 64times this command needs about 58mSek. this is to long....

We are talking about the speed I need.....
2Khz x 4Byte =8kByte per second or
2Khz x 4Byte *60= 0.5Megabyte per minute
I used a Kinston class 10 Card with 45MB/s

I need I good idea to speed up the writing to disk.

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by Benj »

Hello,

It sounds like the 64 sector is the cluster boundary and at which point the FAT routine needs to jump to the File Allocation tables and hunt for and assign a new unused sector address.

If you didn't need the file system FAT support and just wanted to use the card as a data storage then this problem would go away.

I'll have another look at the code to see if I can make any further improvement on the FAT housekeeping but I can't promise anything as I wrote the code originally to be as efficient as possible.

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by Benj »

Maybe also you could use a circular buffer component to store some of the data allowing you to maintain and catch up from the new cluster delay.

stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Yes it's every 64times that is going long.
Unfortunately I need the FAT. A ringbuffer is not so practical, the 18LF6722 has "only" 3936 Bytes RAM.
I hope it's possible to do some improvement of the speed. I think we are close that to the 2Khz goes ....
Anyway the FAT System from you is very comfortable to use and easy to understand.

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by Benj »

I've taken out a double check routine from the SendBuffer function.

I've also got rid of a buffer read when calling the WriteFileSector macro so the buffer will no longer already contain the sector data after a write.

Both of these should give you a little speed boost for your application but otherwise are not recommended for general use.
FAT.fcpx
(66.57 KiB) Downloaded 420 times
Let me know how you get on.

Regarding the circular buffer, I thought maybe you could just use this as a stop gap to overcome the additional 64th sector delay, after the delay you stream your data to your non writing buffer, when the buffer is full you store to the circular buffer until the write is complete, then stream to the card from the buffer and from the circular buffer before resume your alternate buffer functionality. Just a thought, maybe complicated to implement.

stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Hi Ben

Thank for the FAT. I'll test that. And I hope for the 2channel 2kHz recorder with this older PIC will work.

Soon I start to go on with a 10channel 2Khz recorder. I saw microchip has now ARM® Cortex® from Atmel
As if there are faster (300Mega) and need less power... They have a HSMCI port (High Speed MultiMedia Card Interface).
What do you think I should change?

http://www.microchip.com/design-centers ... am-4e-mcus

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by Benj »

Hello,

I would suggest a ST ARM over an Atmel Arm, We have a new version of Flowcode which will be available shortly which will add these devices.

The 16-bit PICs are also very good especially the EP range of devices (70 x 16-bit MIPs vs your current device at 8 x 8-bit MIPs), the PIC32 I am yet to play more with but again these are pretty fast devices.

A standard 8-bit AVR clocked at 16MHz will run at pretty much double the speed of your current 18F PIC device at 32MHz.

stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Hi Ben
"We have a new version of Flowcode"
That nice to hear. Is this FC8 or FC7.2 ? And when can I have it?
Is there a newsletter about FC or general?

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by Benj »

Hello,
That nice to hear. Is this FC8 or FC7.2 ? And when can I have it?
It's 7.2 and it's being beta tested at the moment. We have uncovered some problems which we are currently working on. So coming soon, hopefully the next couple of weeks.
Is there a newsletter about FC or general?
There certainly is you can sign up for it here.
http://www.matrixtsl.com/mailing_list.php

stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Hi Ben
Thank You for the special FAT.It's working faster and I can use my 2channel recorder.

I have now a another small problem to write in a corect way the data to the wave file.

before with CCS I dit it like this to convert the 10Bit uint from the AD-converter to little Endian signed16Bit. But how can i do it in FC7?

ADCIn2=(ADCIn2<<3);
ADCIn2=(ADCIn2-0x1000);
highByte1 = make8(ADCIn2,1);
lowByte1 = make8(ADCIn2,0);
lowByte1 |=0x07;

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by Benj »

Hello,

You can do it like this.

ADCIn2 = ADC::ReadAsInt()

ADCIn2=(ADCIn2 << 3)
ADCIn2=(ADCIn2 - 0x1000)

highByte1 = ADCIn2 >> 8
lowByte1 = ADCIn2 | 0x07

stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Hi Ben
perfect! Thank You. And here a sinus 200Hz Signal sampled with 2kHz and saved as data like a wavefile on a micro-sdcard and open with Audacity (import raw).
recording.PNG
(40.05 KiB) Downloaded 319 times

stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Hi Ben

Now I have to begin with the 10channel recorder. For this project I try to use the chipkit wi-fire with the PIC32MZ2048.
The board include a micro-SDcard connector. I tested the card but it's not fast enough. Is there a possibility to to increase the speed.
This Pic has much more ram. Maybe we can write a cluster 64x512Byte=32768 in one go?

stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Hi Ben

A long time ago you changed the fat for me.
I've taken out a double check routine from the SendBuffer function.
I've also got rid of a buffer read when calling the WriteFileSector macro so the buffer will no longer already contain the sector data after a write.
Both of these should give you a little speed boost for your application but otherwise are not recommended for general use.
But I do not know if I have a new FAT again and again with the FC8 update... I always want the latest update. That would be good
Can you make the "normal" FAT faster for all users?

or another idea would be to have 2 buffers for the FAT.
So I could fill buffer A while buffer B is written to disk and then fill buffer B while buffer A is written to the disk. At the moment I do it like this but I have alway to copy to the writebuffer from the FAT.

Or maybe you some bether ideas.....

stefan.erni
Valued Contributor
Valued Contributor
Posts: 654
Joined: Fri Aug 19, 2016 2:09 pm
Location: switzerland
Has thanked: 182 times
Been thanked: 179 times
Contact:

Re: FAT SD-Card is nice to use, but (very) slow

Post by stefan.erni »

Hi Ben

There is a small problem after 64 sector. The card is like making a break for the cluster.
It sounds like the 64 sector is the cluster boundary and at which point the FAT routine needs to jump to the File Allocation tables and hunt for and assign a new unused sector address
So I made 2 big buffer (64512 Byte), while the card make his break I store the date and write it after the break.....
I have no more errors in the file and I can write in 1 minute a 3MB file (10hours for 1.8GB) My next step is test this 10hours.....

Post Reply