Mapping my Understanding of SRS to Pleco Flashcard Settings

[Jim Kay]

This image represents my understanding of SRS (as Sebastian Leitner devised it.):

You only EVER test yourself on the cards in column P-1. New cards go to Box1 and if you get a card right in any row, it goes to the right-most P-n of the row below it (Box5 cards gotten right go to the Archive and are not involved in any more testing.)

If you get a card wrong in any row, it goes back to the Box1 row.

The thing that makes this SRS the 'spaced' part of it is this: you choose your interval and then the computer moves all the cards in each row one P-n column to the left. So if your interval is 3-days, then the next time you start your program, if 3 days has elapsed, the computer goes through and moves the cards of each column, one column to the left. [it's nice to keep a limit on how many cards can sit in P-1 so I like to let them back up in P-2 and be moved from there oldest-first and only enough to put a maximum of 30 in P-1. Box1 cannot be limited because all wrong cards must go there. I also like having an option for on-demand moving instead of a fixed interval.] [I also like an option to purge Box1 and all of the P-2s of any excess over 30 cards, newest-first, putting them back in the archive.]

In other words: if you get a card right 1 time, you will see it again after one interval. If you get it right a second time in a row, you will see it again after 4 more intervals. If you get it right a third time in a row, you will see it again after 7 more intervals. Then 13 intervals and then it's archived.

There are other nice things to do like taking cards from the Archive (oldest first) and filling Box1 to a maximum of 30 cards. (I have read in several places that 30 is a good limit for one study session.) New cards added to the collection can go to Box1 or to the Archive as the user prefers.

[With three sides, there are six ways to study and each way is tracked indepently, according to the above.]

I know that all of this is feasible because I started with an open-source program on the CodeProject and created exactly this.
=================
So my question becomes: what Pleco Flashcard settings will get me as close as possible to my understanding of SRS?

If I have to export cards from Pleco (which is an outstanding study aid in any event,) there are a number of incompatibilities in card content I will have to resolve.

 

[Cyrano]

Very good question! Plecos srs is impossible for me to understand even after having used it for a year plus

 

[Jim Kay]

I have another flash card system I devised mainly myself (based on a version I got from 'The Code Project" web site. My main study list has 1565 cards but no pinyin and no simplified characters so I spent most of yesterday and today writing and debugging a program to convert my data to Pleco format. I was surprised to see so many 'duplicate' cards listed as I've eliminated duplicates pretty carefully. (Pleco shows me a card with simplified and traditional characters and says it's the same as another card with traditional and Zhuyin. I'm not really sure what was being compared to what so I just took them all.

I knew my cards has some errors in them due to my own and a contract person's typing. So I wrote my program to do a lot of verifying. There were about 2% errors. Better then I expected and whatever I decide with Pleco, at least my data is cleaned up.

I keep reading the Pleco manual, but I know even less than you do about what is happening. (At least it's good for a laugh.)

 

[Jim Kay]

Far_East publishing in Taiwan has a 3000 Most Common Words book with two-character examples for each word and usually one phrase. It has about 17,000 entries in all, and some are clearly duplicates. I have typed i around 8,000 so far. I might test with Pleco optical recognition to see what that does. (I hesitate because the typing experience seems valuable too.)

If I actually finish this list, I'll probably post it.

 

[mikelove]

If you pick the "Spaced Repetition" profile, the default settings in that will pretty much give you this behavior, but we do it in a more fine-grained way - instead of separate boxes, each card has a particular "repetition interval" (its "score"), which gets adjusted based on your correct / incorrect answers. This is pretty much by its very nature harder to understand than the box-based method you're talking about, and in fact our system used to work more like what you're describing, but a number of very popular flashcard apps in recent years (Supermemo, Anki, Mnemosyne, Skritter) have all gone for the more fine-grained interval approach so back in 2008 we switched over to that.

The biggest change in the default settings from what you're describing is that intervals increase exponentially rather than linearly - might go from 1 day to 3 days to 10 days, e.g. If you'd rather have them increase linearly, you can go into Flashcard Testing / Scoring and set Scoring system to "Manual," then in "Configure Scoring" set the score to "increase" "by" "300" "points" if correct "1" "in a row" - that will boost the number of days between repetitions by 3 every time you answer the card correctly.

 

[Jim Kay]

In fact, the box method does not increase the intervals linearly. The interval unit is selectable; then the first time a card is correct (initially or after an error,) the delay becomes 1 interval. The second time correct in a row, the next interval become 4 units; then 7 and finally 13. The 5th time in a row the item is correct, it's considered "learned and it goes to the archive. According to Leitner's research, he asserts these specific numbers are critical! If at any point in the march to the archive, an item is scored as wrong, for interval purposes it it treated as a new item (times correct/incorrect are preserved.)

It appears to me that only a lookup function can produce those exact numbers.

Adding items all with the same score as items just missed can be a problem because the previously missed items should have higher priority.

Finally, there should be an optional method to bring items out of the archive to review again. The number in New status should not be pushed too high and the selection from the archive should be based on some rational criterion.

 

[Jim Kay]

I should have written above "adding NEW items all..."

 

[mikelove]

In fact, the box method does not increase the intervals linearly. The interval unit is selectable; then the first time a card is correct (initially or after an error,) the delay becomes 1 interval. The second time correct in a row, the next interval become 4 units; then 7 and finally 13. The 5th time in a row the item is correct, it's considered "learned and it goes to the archive. According to Leitner's research, he asserts these specific numbers are critical! If at any point in the march to the archive, an item is scored as wrong, for interval purposes it it treated as a new item (times correct/incorrect are preserved.)

It appears to me that only a lookup function can produce those exact numbers.

You can configure it to increase the score by a percentage instead of linearly, so you could probably get awfully close with the manual scoring option. But if you want something that perfectly matches Leitner you're probably going to need to get a different flashcard program - there's hardly any interest in / demand for that now, all everybody seems to want is for our app to work more like those other apps I mentioned and their kin. And actually that manual screen is likely to go away in our next significant flashcard update, along with the 'Weighted' repetition algorithm and a few other seldom-used but tricky-to-maintain features.

Adding items all with the same score as items just missed can be a problem because the previously missed items should have higher priority.

That can be dealt with by lowering the minimum score setting below 100 - new cards will start off at 100 but incorrect cards can then fall below that.

Finally, there should be an optional method to bring items out of the archive to review again. The number in New status should not be pushed too high and the selection from the archive should be based on some rational criterion.

The preferred approach now is not to archive cards but simply to space them out for even longer intervals - Leitner's system predates the computer era, and it didn't really make sense back then to have boxes spaced at 200 units, but with a computer that's no trouble at all.

 

[Jim Kay]

There are so MANY things that are "no trouble at all" to implement on a computer BUT, deciding which of them actually ought to be implemented; well that's a horse of a different color.

I'm comfortable with arithmetic, and I did notice percentages might be a solution, but so far, I have not gotten sufficient understanding of what Pleco is actually going to do so I'm not ready to try selecting this approach just yet.

Before I went down this relatively expensive road (buying an iPad mainly to run Pleco,) I tried to understand how its teaching approach was designed and to what extent it's informed by language educators who actually understand the unique problems facing adults seeking to learn an additional language.

The initial descriptions of Pleco don't offer any information like this and your comment about '...all everybody seems to want...' confirms my impression that Pleco's design is market driven and actual educational effectiveness is not an important concern.

Pleco is really very good for what it does and the iPad does a few other things much better than my PC's do them. (Would you believe even Microsoft, the new owner of Skype, has created a vastly better Skype application for the iPad than they have for Windows 8!) So I am in now way sorry I now own an iPad. I can thank Pleco for pushing me into it.

I've spent the last few days writing and debugging a program to convert data exported from my Flashcard program into the Pleco import format AND, in the process scrubbing all of my data for errors created either by me or by others from my early learning period when I paid people to type vocabulary data for me. About 10% of my data was defective. So that exercise was valuable as well.

Next, I will want to write a program to convert Pleco export data into the format for my program. After that, I'll probably tinker with the Pleco flashcard system, but I have serious doubts that it can easily be made to conform to modern educational research results in the area of adult language acquisition.

I thank you for your dedicated efforts to help me and others get the most out of Pleco. Compared to the cost of the iPad, Pleco is cheap enough, despite its limitations. I'll certainly continue to use it and I expect to have some word lists/dictionary lists to contribute here in the future.

Jim

 

[mikelove]

Before I went down this relatively expensive road (buying an iPad mainly to run Pleco,) I tried to understand how its teaching approach was designed and to what extent it's informed by language educators who actually understand the unique problems facing adults seeking to learn an additional language.

The initial descriptions of Pleco don't offer any information like this and your comment about '...all everybody seems to want...' confirms my impression that Pleco's design is market driven and actual educational effectiveness is not an important concern.

Sorry if it came off that way - the fact is that people prefer these flashcard apps because they work; this approach wouldn't have been adopted by (more or less) every popular flashcard app if it wasn't effective. There's a great deal of research supporting it too - SuperMemo's website has a ton of it, e.g. Educational effectiveness is certainly a concern, but the type of flashcard algorithm that we use is the one that's best proven itself in the real world - it's popular because it works.

But repetition spacing algorithms aren't the sort of thing that are thought up by language educators anyway, honestly - they're more of a subject for cognitive researchers. Inasmuch as there's a Pleco "teaching approach," it's more in the larger philosophy underlying its design, like learning from the living language rather than in the abstract (which is why we make it so incredibly easy to create a flashcard from a new word that you encounter).

Next, I will want to write a program to convert Pleco export data into the format for my program. After that, I'll probably tinker with the Pleco flashcard system, but I have serious doubts that it can easily be made to conform to modern educational research results in the area of adult language acquisition.

Leitner's work was 40 years ago, so I question whether it really is "modern educational research results" - however, if you can show me a current, peer-reviewed study that concludes that Leitner's specific sequence of repetition spacings beats out the approach of of our app / SuperMemo / Anki, we will be more than happy to try to add support for it in our next flashcard update. But absent any research suggesting that Leitner's precise system is superior, I'm inclined to continue with the spacing system that we're using now.

 

[johnh113]

You can configure it to increase the score by a percentage instead of linearly, so you could probably get awfully close with the manual scoring option. But if you want something that perfectly matches Leitner you're probably going to need to get a different flashcard program - there's hardly any interest in / demand for that now, all everybody seems to want is for our app to work more like those other apps I mentioned and their kin. And actually that manual screen is likely to go away in our next significant flashcard update, along with the 'Weighted' repetition algorithm and a few other seldom-used but tricky-to-maintain features.

Dear Mike,
I hope what you mean by "go away" is that it won't be so visible to novice users but will still be available to people that use it. I use it to increase the score by 100% every time I get a right answer, i.e. to double the interval every time I get the card correct. I also set my score to 50 if I get the card wrong. Then when I get it right after 512 days I consider it learned. These settings work pretty well for me. I'd hate to see the manual go away, but I suspect you would provide something equally useful.
John

 

[mikelove]

Dear Mike,
I hope what you mean by "go away" is that it won't be so visible to novice users but will still be available to people that use it. I use it to increase the score by 100% every time I get a right answer, i.e. to double the interval every time I get the card correct. I also set my score to 50 if I get the card wrong. Then when I get it right after 512 days I consider it learned. These settings work pretty well for me. I'd hate to see the manual go away, but I suspect you would provide something equally useful.
John

We had actually been planning to eliminate it altogether, though it's not quite as critical that we get rid of it as that we get rid of "Weighted" (which really complicates some other things we'd like to do) so it may stick around albeit more buried than before.

What you're describing would actually be quite doable with our automated system: in the Tweak Parameters screen, you'd set "Easiness Change" for every quality to 0 (so the easiness always stays 100), "Initial correct score" for all three options to 200 (so it'll double from 100 to 200 after your first correct answer), "correct scale score increase %" to for all three to 100 (so that feature will be disabled), "minimum score" to 50, and "easiness divisor" to 50. That way, every correct answer will double the card's score (100 easiness / 50) and every incorrect answer will set it back to 50, just as with your current manual settings.

Most of the possible configurations in Manual can also be achieved through Tweak Settings, so in general I think we'd rather add one or two additional options to that than continue on having multiple relatively-similar scoring systems.

 

[gato]

Wow, I need the "manual scoring" screen because spaced reptition isn't right for the way I use the cards. I hope that'll still be in the next version.

 

[HW60]

I usually use SRS, but I do not like it very much because I have no control about the number of cards I want to test. I could use filters or reduce the number of categories or just stop testing before I finished the required amount of cards, but I still feel that I have no controll about the test in that case.

I therefore use Frequency adjusted testing from time to time, where I can tell Pleco to test 30 cards which (hopefully) are selected by some weighted algorithm. This is still an Automatic Scoring system. Is it that one you would like to go away?

I would prefer a scoring system which combines SRS and Frequency adjusted testing, selecting the cards that are due according to SRS, present them in a frequency adjusted order, and stop after I did the number of cards I would like to test.

 

[mikelove]

Wow, I need the "manual scoring" screen because spaced reptition isn't right for the way I use the cards. I hope that'll still be in the next version.

How so? As I said, we're looking to add a few options to Automated to make that cover these sorts of scenarios. This is about how the scores are calculated, not whether or not you have to use SRS - is there a particular score calculation achievable with Manual and not with Automated that you'd like to continue using?

I usually use SRS, but I do not like it very much because I have no control about the number of cards I want to test. I could use filters or reduce the number of categories or just stop testing before I finished the required amount of cards, but I still feel that I have no controll about the test in that case.

I therefore use Frequency adjusted testing from time to time, where I can tell Pleco to test 30 cards which (hopefully) are selected by some weighted algorithm. This is still an Automatic Scoring system. Is it that one you would like to go away?

I would prefer a scoring system which combines SRS and Frequency adjusted testing, selecting the cards that are due according to SRS, present them in a frequency adjusted order, and stop after I did the number of cards I would like to test.

The plan for Weighted / Frequency-adjusted testing is essentially to replace it with very short intervals in Spaced Repetition, as several other popular systems do now - an incorrect card might (say) come up again in the test 10 minutes later. This way, you get the continuous review of Weighted but also the assurance that every card ends up getting reviewed, something you don't get with our current randomized Weighted system.

But honestly, combining SRS with a set number of cards simply doesn't work; if you don't keep up with the cards you have due for review on a particular day, you're going to forget lots of words, because you won't have reviewed them often enough. We do, however, plan to add some options to make it easier to bury some cards if you have too many due for review.

 

[HW60]

The plan for Weighted / Frequency-adjusted testing is essentially to replace it with very short intervals in Spaced Repetition, as several other popular systems do now - an incorrect card might (say) come up again in the test 10 minutes later. This way, you get the continuous review of Weighted but also the assurance that every card ends up getting reviewed, something you don't get with our current randomized Weighted system.

But honestly, combining SRS with a set number of cards simply doesn't work; if you don't keep up with the cards you have due for review on a particular day, you're going to forget lots of words, because you won't have reviewed them often enough. We do, however, plan to add some options to make it easier to bury some cards if you have too many due for review.

My problem is that I spend too much time with Pleco and too little time with books and other tools to learn. Therefore I want to reduce the amount of cards on some days. The "assurance that every card ends up getting reviewed" is just the problem I want to solve, and actually I am responsible for my knowledge of cards, not Pleco. It is already now that I am "going to forget lots of words", because Pleco has no support for such days when I would like (or have) to reduce the time used for flashcards.

It is a great difference if I do not review a card which I did not know the day before - these are the important cards - or if I do not review a card that is due after e.g. 100 days - 5 more days waiting will not hurt in that case. And frequence adjusted should just select the important cards out of the cards that are due according to SRS.

When I cannot review for 2 weeks, SRS presents me 3000 - 4000 cards to review without support. The problem would at least be a little smaller with the SRS+Frequency adjusted working.

But actually I would not be very happy if you changed Frequency-adjusted, because now after much trial and error I found a manual solution for SRS+Frequency adjusted, and I would not like to search for another manual solution. And I would never spend the time to use Anki-like systems with "review in 2 minutes" or "10 minutes" or else - one session a day is already more than enough.

 

[mikelove]

My problem is that I spend too much time with Pleco and too little time with books and other tools to learn. Therefore I want to reduce the amount of cards on some days. The "assurance that every card ends up getting reviewed" is just the problem I want to solve, and actually I am responsible for my knowledge of cards, not Pleco. It is already now that I am "going to forget lots of words", because Pleco has no support for such days when I would like (or have) to reduce the time used for flashcards.

But that's what I'm suggesting we'd like to add - a way to reduce your number of cards due more intelligently by removing some cards from active review, so that you'll study the remaining cards often enough to continue retaining them. Thus maximizing the number of cards you can learn / retain in the time you have to study. If you only study the highest-priority cards every day, the number of overdue cards is going to keep piling up until eventually even the highest-priority ones are being studied a week late.

It is a great difference if I do not review a card which I did not know the day before - these are the important cards - or if I do not review a card that is due after e.g. 100 days - 5 more days waiting will not hurt in that case. And frequence adjusted should just select the important cards out of the cards that are due according to SRS.

That's a reasonable point, but the overdue pile will still build up even if we prioritize intelligently - if you're only studying half as many cards per day as you need to, you're going to end up studying those 100 day cards at 200 days.

But actually I would not be very happy if you changed Frequency-adjusted, because now after much trial and error I found a manual solution for SRS+Frequency adjusted, and I would not like to search for another manual solution. And I would never spend the time to use Anki-like systems with "review in 2 minutes" or "10 minutes" or else - one session a day is already more than enough.

We can't keep around features that don't work very well forever. The flashcard system is simply too complicated now, and features like frequency-adjusted are a big part of the reason why; one scoring system and one card selection system ought to be more than enough.

 

[HW60]

If you only study the highest-priority cards every day, the number of overdue cards is going to keep piling up until eventually even the highest-priority ones are being studied a week late.

if you're only studying half as many cards per day as you need to, you're going to end up studying those 100 day cards at 200 days.

We can't keep around features that don't work very well forever. The flashcard system is simply too complicated now, and features like frequency-adjusted are a big part of the reason why; one scoring system and one card selection system ought to be more than enough.

I solved the typical SRS problem of too many due cards (due to a review break or due to over optimistically adding too many new cards) by manually creating an additional frequency adjusted (FA) profile. Then I test as many categories in the SRS profile as I like to per day (approx. 300 now) and move the other categories - that belonged to the SRS profile before - to the FA profile. In this profile I test about 30 cards per day (or more if I like to). Actually the profile has more than 500 cards, so I only test about 6%! The idea is not to learn, but not to completely forget cards I had already started to learn. Then I move the categories one by one back to SRS as soon as possible, looking at the maximum of 300 cards again.

In the past I only reduced the categories of the SRS profile and did not look at the other cards I had no time to review with the result, that now I sometimes transfer cards from existing categories or even from uncategorized that were reviewed more than 900 days ago and which are like completely new cards for me.

I cannot use a second SRS profile for the temporarily removed cards (and stop testing after 30 cards) because the selection of cards would then be purely random, but I want to repeat the important cards, selected by weighted frequency.

I do not know if your planned new flashcard system solves the problem of "maintaining some knowledge about old flashcards" as described above. I do not even know which of the actual systems you want to keep and what you mean with "that manual screen is likely to go away" which made some people like me nervous: There are 4 card selection systems and 4 Scoring systems (only in "Scoring system" a "Manual" value can be chosen).

 

[mikelove]

In the past I only reduced the categories of the SRS profile and did not look at the other cards I had no time to review with the result, that now I sometimes transfer cards from existing categories or even from uncategorized that were reviewed more than 900 days ago and which are like completely new cards for me.

I cannot use a second SRS profile for the temporarily removed cards (and stop testing after 30 cards) because the selection of cards would then be purely random, but I want to repeat the important cards, selected by weighted frequency.

The selection of cards wouldn't be purely random even in the current system - it shows you the highest-priority ones first. But it does this in a more organized way than it would with a weighted random test. And what we'd like to do is accommodate those in SRS more officially - remove some cards from active review while you catch up, but then prioritize bringing them back once you're ready for some new words.

I do not know if your planned new flashcard system solves the problem of "maintaining some knowledge about old flashcards" as described above. I do not even know which of the actual systems you want to keep and what you mean with "that manual screen is likely to go away" which made some people like me nervous: There are 4 card selection systems and 4 Scoring systems (only in "Scoring system" a "Manual" value can be chosen).

Basically, the only remaining card selection options would be Random and SRS - "Fixed" has had a dagger over its head for a long time and we think we can kill it officially if we just offer a slightly nicer viewing option in Organize - and the only remaining Scoring System options would be None and Automated. (and probably Stats Only since that still has some practical use) A number of other options would be streamlined or eliminated too.

We would probably debut these cuts on Android first, since unlike on iOS, on Android we'd be able to offer people a way to revert to the old version if they didn't like the changes. (while we went through and added solutions for any popular use cases that the new system actually didn't cover before making the same cuts on iOS) But we simply can't make our flashcard system all that we'd like it to be as long as we're tethered to all of these old, rarely-used features - we have to make some serious cuts.

 

[gato]

How so? As I said, we're looking to add a few options to Automated to make that cover these sorts of scenarios. This is about how the scores are calculated, not whether or not you have to use SRS - is there a particular score calculation achievable with Manual and not with Automated that you'd like to continue using?

But honestly, combining SRS with a set number of cards simply doesn't work; if you don't keep up with the cards you have due for review on a particular day, you're going to forget lots of words, because you won't have reviewed them often enough. We do, however, plan to add some options to make it easier to bury some cards if you have too many due for review.

The Space Repetition algorithm requires being able to study every day and doesn't work for people who only study sporadically. If you don't study for a day or two (or more), you get swamped with cards to review.

Spordic study is the reality for many people, though it'd certainly be better to study more regularly.

I use the "manual scoring" and "fixed order" to make the flashcard system work like physical flash cards.
See my posts about this here:
http://plecoforums.com/threads/best-way-to-use-flashcards.3150/#post-25483