FAT problem

A forums to allow bugs and problems with Flowcode v6 to be reported and resolved.

Moderator: Benj

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

FAT problem

Post by mikn »

Please, review the project.
It takes ADC data at 8khz frequency and stores into data.dat file.
I am using 16gb class 10 sd card.
pub.fcfx
(15.55 KiB) Downloaded 285 times
The problem is that every 4th cycle of FAT write the length of this cycle rises. (I have one big cycle which includes 4 small, write 2048 bytes as 512*4, but the problem appears every 4th BIG cycle, not small)
If you can see in the project, i have LED ON when it writes and off when finished. In real world the problem looks like this: 3 fast blinks and 1 slow.
From the beginning after I power on the hardware everything looks ok, each blink is same length. But after some time I note that each 4th blink is longer and longer.

I loaded the data into excel table to see what's wrong and I see that starting from approx. 20.000 sample I have skipped data (I implemented a flag to skip buffer overwrite while FAT is flushing it). After that the buffer pauses each 16380-16390 samples.
problem.GIF
problem.GIF (9.08 KiB) Viewed 13491 times
The problem for me looks like the FAT is writing data slower and slower, but I wonder, why it do this every 4th cycle and not each cycle. The SD card is enough quick to write data much faster than this.

Please, check what may be wrong since I am stuck in the project and can't finish it now because of this bug.
FC 6.1.3.2 (18.02.2016)

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Today I left the device to write data for the whole night and in the morning it was writing with big delays.
Also the problem continues even I do power off and power on. It must delete the file and start from the beginning, and it did so but started writing with big delays.

I noted that after full night work when I inserted SD card into my PC it detects too long time (around 15 seconds instead of 1-2 seconds when it's blank or filled with other files).

Aren't there any issue with FAT table which FAT component generates after creating the file? Because once I format the card it start writing normally, but after some time delay increases continuously again.

PS: I've changed 5 cards different class, capacity and make but everywhere there's same problem.
FC 6.1.3.2 (18.02.2016)

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

here's the video of led flashing

with blank formatted card:
moviefrm.rar
(148.61 KiB) Downloaded 326 times
and after couple of hours:
movie 1.rar
(344.97 KiB) Downloaded 340 times
FC 6.1.3.2 (18.02.2016)

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Please, give me any kind of response about the issue. Either "we've located the problem and need 1 week to resolve" or or "we need 1 year to resolve" or "it's your problem" or "we don't work with FAT anymore" or "just shut up and go ahead, you dumb" or at least something so I could count on something. I've bought PRO version and almost completed the commercial project for the client and now it's a big problem that I can't release it.
FC 6.1.3.2 (18.02.2016)

hyperion007
Posts: 528
Joined: Sat Dec 01, 2012 1:23 pm
Location: Sweden
Has thanked: 49 times
Been thanked: 101 times
Contact:

Re: FAT problem

Post by hyperion007 »

Hi Mikn,

Could you please try this version of your program and see if it works any better.

Let us know if or how it worked.

/Daniel S
pub.fcfx
(16.93 KiB) Downloaded 328 times

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

ok, will try in the evening.

but in your case I will have dropped measures during FAT flushing, because the INT will be stopped and not capturing any data from the port. and I need non-interruptible measures.
FC 6.1.3.2 (18.02.2016)

hyperion007
Posts: 528
Joined: Sat Dec 01, 2012 1:23 pm
Location: Sweden
Has thanked: 49 times
Been thanked: 101 times
Contact:

Re: FAT problem

Post by hyperion007 »

Ok, then could you perhaps change your program to create a new file on the SD card after for example 5 flush cycles. I think that it takes longer and longer to write the data because it takes longer to find the end of the file where you append new data the larger the file gets. Then when you have sampled enough data you could have a routine that takes each small file and append them to data.dat one by one and afterwards deletes all the small files.

You can test the theory by having the current program run for awhile until it starts running slow. Leave the file on the SD card, take it out to be sure it's not overwritten or deleted. Then make a simple while 1 loop an put it at the top of your program. Here you make a routine that starts a timer, then opens the file from the SD card, then stops the timer and prints out the time it took to open the file.
Then do the same test but now have the program to append a single byte to the file. It should take roughly the same amount of time as it did just to open the file.

You might want to crank up the speed of the PIC as well, to make it work as fast as possible. Might eliminate some issues or just move up the limit of when it starts falling behind.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

to create a new file on the SD card after for example 5 flush cycles.
yes i did so, in original code i created file every 5 minutes. but the problem was accumulating. doesn't matter how much cycles i pass before creating new file, each new file the pause makes longer and longer. so seems like the FAT component has some issue with the whole fat table.
FC 6.1.3.2 (18.02.2016)

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Sorry for bothering you but I just wanna know if you located the problem with the FAT and when do you plan to have it updated?
I am stuck with the project. :cry:
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT problem

Post by Benj »

Hello Mikn,

Just to save me trawling through all the posts and potentially missing things could you create a list of all the specific problems your experiencing with the FAT component and I will aim to have a good look at the code this afternoon for you.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Benj, I have described the problem in the first message. This is the only problem. Upon filling the SD with data the write cycle time rises. I can't do realtime measure of analogue input and save the data to SD. Writing cycle becomes slower and slower. Even I create new file, it's still slower and slower. but after I format the card - everything is ok and starts from the beginning: fast write and then slower and slower.

I assume that problem depends on the FAT size. the more files and data are written to SD the slower it works. like it reads all fat before each write cycle and it takes more and more time each time.

the target I need: to do realtime measure of analogue input at 22khz and write data to SD (class10)
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT problem

Post by Benj »

Hello Mika,

How are you storing the file? If your opening the file each time or using the simple append string function then yes I would imagine this will get slower and slower as the file grows in size. Edit I've checked this and the append function shouldn't involve longer and longer delays as long as you don't keep re-opening the file.

If your using the direct method of writing sectors to a file then I think this should remain at a fairly constant rate. This is what I used to create the ADC data logger example back when I originally wrote the component.

However when assigning a new cluster to the file (probably every 32 sector writes (32 x 512 bytes)) there will be a bit of an additional delay as we have to search about for the next free cluster to use. To get around this additional delay you might need to say use a circular buffer on an interrupt to collect the data and then in your main while there is data in the circular buffer attempt to write it to the card. This way you maintain a constant sample rate and writes to the card happen as and when possible.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Benj, check the project in my first message. I use writefilesector. The code is simple. And this code give me delays. I also recorded a video how it works. Please, spend 10 minutes to check what I have posted. I give not words, but real evidences of what's going on at my side.
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT problem

Post by Benj »

Hello Mikn,

Sounds like on your card each cluster is made up of 16 sectors, finding a new available cluster is causing your timing issues. I've now seen the vids and the delay does seem to be very long. Hmm I wonder if there is an improvement to be had with the way we check for the next free cluster.

Please can you try this file and see how you get on. This should sample on a steady basis and then write to the FAT as soon as there is over a full sector of data available in the buffer. There is also a clear indication now if sampling hits a problem.
pubBR.fcfx
(14.87 KiB) Downloaded 183 times
Let me know how you get on.

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT problem

Post by Benj »

Looking at the code that searches for the next free cluster it does always start from the beginning of the FAT. I'll work on it and see if I can improve this.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Here's the result: many fast blinks and one long.
Video included.
Attachments
pubBR-result 1.rar
(58.33 KiB) Downloaded 186 times
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT problem

Post by Benj »

Hi Mikn,

That looks slightly better.

Here is a slightly modified component which starts looking for the next free cluster from the current FAT sector rather than starting right at the start of the FAT. Means we are much less likely to fill the card to 100% but should help to maintain speed.

Fingers crossed this combined with using the circular buffer will help to solve the issue.
FAT.fcpx
(62.79 KiB) Downloaded 175 times

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Ok, I will start the test now for several hours and will inform you about result closer to the evening.
FC 6.1.3.2 (18.02.2016)

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

After around 1 hour it started blinking 1 second on your control point. So the problem continues.
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT problem

Post by Benj »

Hello,

That's very odd, I wonder what could be causing this.

I suppose the best solution is for me to setup the hardware and see if I can replicate the problem and then dig around and see if I can find out where the delays are coming from. Finding some free time to devote to this might take a while.

So...

Increasing the size of the circular buffer component would likely give you longer but this obviously is not ideal and the case might be there is no more memory to spare.

Can you increase the speed of the microcontroller at all? Doing this would give you faster SPI speeds as well as more overhead for transfers between interrupts which might get rid of the issue for you completely.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Now it's at 16mhz. How to increase it to 32 using FC? The oscillator selected in config is FRC with PLL, but I think it's not operating at it's maximum speed.
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT problem

Post by Benj »

Hello,

If you look at the device datasheet, specifically page 99 detailing the CLKDIV register then it details what is required.

The internal osc is currently set to 4MHz x 4 = 16MHz. This can be speeded up to 8MHz x 4 = 32MHZ.

So I would recommend adding the following code in a C icon at the start of your main and also upping your clock speed to 32000000 in the project options. Fingers crossed this will make everything work as expected.

Code: Select all

CLKDIV = CLKDIV & 0xF0FF;

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Did so. It made many fast blinks during 0.5 sec and then stopped at your control point, it's 1sec blinking.

Benj, if you will remember, in FC 6.0.8 or kinda that I've got a problem with Software pin remap. FAT was working very very slow and after you changed something, FAT become working much faster. Isn't the problem somewhere in software mode?
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT problem

Post by Benj »

Hello Mikn,

Did you change the timer interrupt prescaler after increasing the clock speed? If not then this will be working twice as fast and therefore not giving you any advantage.

FAT in software mode didn't used to use the SPI speed up after initialise that the hardware mode did. This fix is now built into the SPI CAL for 6.1 so should no longer be an issue.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT problem

Post by mikn »

Forgot to change it.
Now started working, will inform you after several hours. Or when it will stuck again.
FC 6.1.3.2 (18.02.2016)

Post Reply