News

SimBank or 200 SmartCards in a box (full article)

As I already mentioned in my first Habr's topic about PCBs creation, my firm deals with various telecommunication devices such as VoIP, GSM, PBX gateways, GSM controlled rebooter sockets, etc. Today, I'm going to describe the process of SimBank firmware development in its PCI version where I had the role of FPGA layout developer.


Management tasks background + device development history

 

SimBank unit is designed for SIM (Subscriber Identification Module) cards, a variation of ISO-7816 cards. This unit may be used for centralized SIM cards or other smart cards storage to be used in such devices as GSM gateways, satellite television tuners or other devices which operation is based on smart-cards. Together with SIM-Server software, SimBank establishes a flexible system for recording and control of SIM cards you use in your applications, ensures huge capabilities of setting and configuring the system operation via a user-friendly interface. At the same time, the cards themselves are stored in an easily accessible location and are connected to terminals over TCP/IP.


There are other articles on Habr describing the design and procedure of operation with smart cards:

Brief Introduction into SIM Cards by OgreSwamp
Snart Cards for Little Ones by brake
A Smart Card Design by rlepricon


I read them all with pleasure before starting my own development. And I won't repeat anything you can find in these articles.

 

SimBank or 200 SmartCards in a box

SimBank Design

 

This design was created by my firm long ago and has been used successfully for many years. But there was one specific feature about all the firm's PCI units: a training PCI core that was supplied "as it is" without any claims accepted. Anyway, there were no claims. With just minor redesigning, this core was installed to all the boards without any exception. This helped using the available address space intelligently. With one and the same Vendor ID, Device ID, Class Code. As the units were usually assembled at the firm and passed the functional tests, no conflicts were reported. With some time restrictions, the principle "do not touch if it works" is quite a smart approach. This was Ok until some contemporary mother boards started to show unstable operation or even failed to launch.

 

Time has come to pay more attention to the PCI core. The training core adjusted for specific tasks was documented and commented, but these comments were not always sufficient. The link for its location dated 2002 or 2004 was inaccessible.
The task was more than simple: "The unit must work on all the standard boards."
And a bonus: "Even if not all 200 SIM cards are used, but 196 or 1хх."
Until that moment, I've never worked with SIM cards or PCI bus as a FPGA developer. Just a circuit and a board. I had some experience in VHDL and Verilog.
It was decided to set for the development from PCI as the part related to SIM cards was operational.
The familiarization with PCI Local Bus specification did no miracle and did not cast much light on the path to go. Having read multiple articles about do's and don'ts, but still having no clear picture in mind, I decided: "just start, then I'll figure it out."
There is a Megafunction for Altera PCI in Quartus environment that allows for generating a PCI core with the required parameters. The core is attached with detailed documentation and there is the PCI kit description on the web-site with PCI model of behavioral simulation in ModelsSim-Altera. All this together is a great help for those starting from the scratch.
Moreover, Altera allows using its Vendor ID (1172h).
In PCI specification, a unit class unavailable in earlier specification versions was selected: 07h – Simple Communication Controllers, 05h- Smart Card.
Then, everything should work in Linux. As my colleague was more experienced in SIM Cards and PCI, he was the one to develop the technical specifications. And all my thanks to him. Properly compiled specification is the half of the task. Though, it requires "additional writings."

 

Material studies

 

As it was not me who designed the unit, I had to study its circuit and board layout.
The circuit is necessary for understanding the links and connections and the board helps to know where to plug the programmer, SIM cards and where to look for any service indication. Later, I will tell you about a low-down trick it played with me.

SimBank or 200 SmartCards in a box

For the circuit, we have a board with two Altera Cyclone II EP2C35672C8 FPGAs in a housing with 672 contacts. One FPGA is connected to the PCI bus and controls the second PCI. 100 SIM card holders are attached to each FPGA. Each SIM card is linked with its own reset and data signal, and CLK signal comes to a group of 10 cards.

SimBank or 200 SmartCards in a box

Each EP2C35 microcircuit contains 105 M4K units of built-in memory. Generally, RAM unit (including parity bit) has 4,608 bits.

 

Units support various configurations from 4K × 1 to 128 × 32(36).

 

On top of this, I wrote a document with additional questions about the transmission sequence and various signals duration regulation, and about verification of the incoming data from SIMs and other questions.

 

Having make it clear about the contacts of the first PCI bus from the microcircuit, I made a pilot project of the PCI core in Quartus that was supposed to work with new bus parameters. Without any logic – just a core.
After I received the response to "lspci -vv”" command, I felt myself the happiest man in the world.

SimBank or 200 SmartCards in a box

 

Then we discussed with the driver and application developer our future actions and made a pilot project in which the driver wrote the data to its memory and than read them and checked. Initially, only the first microcircuit in the layout was checked.
Then came the second microcircuit detailing.
36 signal lines + 2 lines between PLL_OUT and CLK output are designed between this two microcircuits. The previous project used 19 address lines and two bytes for the data exchange (one per each side), read signals and write signal. This was determined by the features of the existing core. And by the consideration of the driver and software unification. So, for example, there is an established set of addresses where you can take any service information from.
ARM versions store the following info type:

SimBank or 200 SmartCards in a box

 

 

Elgato G4_1 is the unit type (“K16” or “К32” for PCI cards, “SimBank” for SimBank);
SIM51215 (SIM900 or other) is the type of the installed GSM or 3G module (required to select a proper set of AT commands). Its relation with the described unit is the selection of a SIM card to be connected to the operator's network via the given module.
2014 is the year of development;
ver.14.144 is the firmware version;
SN: 0123

 

If it is not changed to the correct version before the firmware installation, these data may help to learn many interesting facts form the "register" stating the project customer and any deviations from the standard the customer wished.
For SimBank, only a serial number is sufficient as, functionally, there's always just one SimBank card in a unit. And in PCI version, this card is just one for the firm.


Painful Decision

 

Earlier core version used two BARs. BAR1 was mapped in 1М memory, BAR0 was used as 4К input-output registers. All the address space was assigned to both FPGAs simultaneously.
It was decided to refuse IO application for different reasons. First, it is much easier for the driver developer to work with the memory. Second, there are many recommendation in the Internet explaining why it is advisable to refuse from IOs in new developments. Third, the IO resources available for setting in Altera core is restricted by 256. Memory resources are much bigger. But this transition required changing the SIM card operation principle. And implied almost complete re-working of the exchange module for SIM. In addition, moving from a traditional addressing meant we would not be able to read the required bits in a customary location using any of our typical programs.
We could refuse from making two base addresses and get along with just one. We could do without one timing signal saving at exchange with the first microcircuit cards. This data exchange portion is not responsible for voice transfer and we do not need to save milliseconds to get not more than 20 ms delay. Anyway, I believed this would be right.


Structure

 

After the first success and verification of the text data writing into the memory, the time has come to build the structure of data exchange with the SIM cards. I had some general understanding of this process, but after the tests, I was ready to suggest it would work. An the tests I elaborated and conducted turned to be useful.
The existing 36+2 lines were insufficient for broadcasting the service signals of the PCI bus to the second microcircuit without changes, and I was unwilling to restrict the receipt and transfer by one or two bytes. Even if I manage to separate address and data buses.
Therefore, 32 of 36 signal lines were allocated to the data (and address) bus from the very beginning, 1 signal line controlled the general system reset and the other three were loaded up with all the control functions. We had a separate line to transmit CLK signal.
Silent mode, address transfer mode, single data writing mode, data threads (burst) mode, masks writing modes, reading and threads reading. Total of eight bus conditions that we managed to put into three signal lines.
There still were four BE signals to be transmitted for mask writing. We decided to hide them in four higher bits on the data bus if we use such a mode.

 

For each SIM card in the FPGA, a maximum possible buffer was allocated. Taking into account that a single-way half-duplex interface was realized with a smart card, that we had 100 cards and total of 105 memory cells in the microcircuit, it was decided to make just one buffer for receipt and transfer, and reset addressing in the beginning of any receipt or transfer. I was lucky, and it was a standard limitation for a SIM card to send not more than 256 data bytes per a command. Therefore, the available buffer won't get overfilled before we can read it.
Here, "command" meaning is a bit different from that of ISO-7816 standard. The standard takes it as addressing the card, then receiving the confirmation from the card and then data receipt. But I was interested in the data amount between the buffer resettings.

 

It was decided not to make a buffer for the data exchange with the second microcircuit, but to make an immediate record in the SIM card memory in the second microcircuit. Because 102 of 105 memory units were already occupied. The selection between the first and the second microcircuit was made via addressing BAR0 or BAR1, respectively.
Different controllers were provided for BAR0 and BAR1. The second microcircuit is addressed during the next clock period, but this is adjusted by TRDY signal delay setting at the PCI bus.

 

Each microcircuit has 64К of memory which is sufficient for 128 Smart cards processing. We have 100. 28 units are left. So, we locate some service information in two units such as the information about the card resetting condition, the exchange rate. In return, we need to get the number of data received from the SIM card.

 

Then, everything should be made operational and debugged. As a project for 100 SIM cards takes very much time for compiling, and it's much more expensive to burn our 100 SIM than say 1 or 10, we decided to debug exchange between 1 to 5 cards first and then add all the rest. Moreover, to debug only the first half and then, after we decide that everything's ok, to go to the second microcircuit.
It was here where a surprise awaited for me.
There is a circuit with cards contacts numbered 1 to 100. There is a board project with successfully imported FPGA contacts settings (PCI is operational, test memory record works and the exchange between he microcircuits is ok). The project is compiled of 5 cards, to have at least a second card working if I am wrong with numbering from 0 or 1. Not a bit of it!
Ok, we have 10 LEDs on the board. Though I planned to leave them for the end, I had to move my plans a bit closer. We add a simple thing for the LED on-off, but the lamp does not lit LED does not flash.
So, we take an oscillograph. There's a CLOCK, but no reset. No data, too.
Is anything wrong with the data import? Let's check the circuit against the board, then the board and circuit against the Pin Planner in Quartus. They match. But there's still no reset.
It is the moment my more experienced friend comes to help me. He says that “a princess is in another castle” “SIM0” is at the other side of the box in the program. And it always was there.
Here you can screen shot a piece from PCB project to P-CAD. There are signal name plates there.

 

On the board, SIM 199 is used, and in the project, it is SIM0. The project is compiled for 5 cards only. (It is not big enough yet for 100.) And therefore, the rest 95 cards are just not served. For them, even no FPGA memory buffer is created.

 

To save the resources, it was decided to time the memory units with the frequency of SIM cards. This will help to reduce the number of dividers. Besides, with a lower frequency, it will be easier to fit into the speed resources of the microcircuit. At the same time, it was planned to write the SIM buffer at the PCI bus frequency (33 MHz), and to read it at the SIM card operation rate. Due to the restrictions of two port memory, we had to look for other solutions in Cyclone II. So, the easiest and rightest solutions turned to be timing of all the units with 33 MHz frequency from PCI_CLK. And for reduction of the number of multidigit counters, we had to use permission signals for registers timing. In addition, this approach allowed for easier writing of time restrictions for TimingAnalyzer via multycycle.

 

Model setting. For the success of any invention, we need a model. For the PCI core, this model is represented by Altera. For Smart cards, we had to write a model ourselves.
The disadvantage of this solution is if I understand the SIM card operation wrong, I will write equally wrong model. But it worked. Though, not from the very first time.
Simulation helped me not to become a nuance for my colleague, though afterwards, we had spend much time solving the hardware problems while debugging. What is wrong in the model and how it should be.


T0/T1 or never try to make it better, make it right

 

Initially, in automatic SIM card operation, the SIM card resetting and reading from ATR was debugged together with rate change and a simple command "A0 A4 00 00 02" sending with the response of "A4" for all of these. I wanted to make an automaton that reads everything itself at switching on and operates at a maximum SIM card rate by the moment of OS launching. This would help to unload the program when initializing two hundred (200!) SIM cards at the start.
Some SIM cards worked nice, but others were stubborn returning h’3B and staying silent. As a result, a timer was tripped automatically and I reset the SIM card as it failed to tell me the rate I could talk with it. If I received three or more than three bytes, I believed the ATR was read and I could go on working. Though the model worked perfect, it turned out that different SIMs give different ATRs. Some give all the information at once, while others set a pause after sending a heading of h’3B and only then give the necessary data. Bigger pause solved the problem. For a while. There are SIMs that send a final bit with the same long delay as in the beginning after h’3B byte.
It is solved, too. No smart automatons, spontaneous rate changing over and other things. It's the task of the processor to think of this. It must make a decision about when to give a reset, when to remove it and decide when the data portion from the card is finished.
But there's a problem. We still did not set all the goodies, and we can fit only 80-85 SIMs out of necessary 100 in each microcircuit. Let the operational system driver do this. It's not a big task for it, and we feel better.
Then we set about debugging other commands we already received from the OS driver. Debugging on a big number of cards. And new "discoveries" awaited for us.


197 of 200

 

Where is the solution if 197 of 200 of cards are operational, but three are not? Is it an accurate operation? We debugged this on SIMs blocked by the operator that still gave responses to resetting and pretended they write SMSs to them. May it be that these three cards are blocked intentionally and they are mistaken? How naive I was.
The first real operation showed everything was correct at their end. I had to search.
And I found. Not soon. Again, a colleague of mine helped who arranged exchange logs for all the SIMs. And it turned out that my counter skipped sometimes when writing big commands. So, I wrote the same byte twice in the middle of a long transfer. As behavioral modeling is a long process, I did not use long commands in the model. Only short transactions. I used reduced counters. A step ahead, model, check, works. A small victory! And so on. A bit is written, then a word, a double word, A couple of addresses are passed – hopefully, everything will be ok.
As a result, it works with a small model, but stumbles in real life. At the same time, the SIM card has the time to give its number (if written in it), ICCID, IMSI or whatever else before the error. And when simulating writing/reading to/from the SIM card, it does not demonstrate utmost stability: three of 200 cards give errors.
Manual comparison of the program log with the logic analyzer showed the error resides in the SMS body. And 197 cards overlook this error when counting CRC. They are unwilling to spend their efforts for SMSs, though three cards celebrate their hour of triumph.
I updated the model and launched it, and it again was ok. How could it happen?
I start the model with complete counters check (divide 33 MHz by 6, then by 372, then so on and so on, instead, suppose to divide by 4 and then by 16). So, we start the model in the evening hoping to see everything in the morning...
and in the morning, we find out OS sleeping calm and quite. The screen used to switched over to energy saving mode. But it never slept. After 4+ hours of modeling of more than 300 signals and buses, it fell asleep "in its shoes". And started only at the third attempt in the morning. Though, I was lucky enough for I was saved from the necessity to restore the whole system.
A day of work, sleep is switched off. And the next morning, I was happy finding the error. To be honest, before, it took me a day to deal with various other tricks. On my happy day's morning, I guessed where to find the error.
It was not the last – there were others, more evident and easier to correct ones.


Laying the mosaic

 

Assuming the smart card operation is stable, I connected the second microcircuit. First, in the project. Then, in the model. Then, in the hardware. Not at once, but we did this portion, too. How to verify all the 200 locations work simultaneously and do not interfere with each other? Where to take so many SIMs? We scraped the corners and found 200 cards. Model was launched.
Launched and set to testing. And a new surprise. If all the positions in a SimBank are occupied, it works good. The cards are stable, resettings are rare. But, if some cards are not installed, a resetting command comes to their positions regularly. And sometimes, the symmetric cards in the neighboring FPGA are reset, too. Why?
I still cannot answer this question. But now, I know what Signal Tap II Logic Analyser means in Quartus by Altera. We used it to retrieve the signals conditions in the PCI bus via JTAG. Changing the project and creating new signals to start or stop counting, we managed to look at some parts inside FPGA. So, for example, I was surprised to know that Burst mode almost never was switched on when reading or writing more than eight bytes. Absolutely never for writing, and not more than two PCI clock periods for reading. Though the program sends command "read 40 bytes", reading is processed by four bytes from setting the address on the PCI

SimBank or 200 SmartCards in a box

Resetting problem was partially solved with program tools at the driver end, partially by restricting the usage of odd number of SIM cards in the bank until better times.

 

A question to the readers: Is there any possibility to write all the logs of a device calling in Linux (Centos 6 32-bit)? It is desirable, though, not to interfere with the PCI bus operation. So that we could set a Vendor ID/Device ID and store its R/W calling logs in the bus?


Conclusion

 

Finally, fully updated project in FPGA spends 89% of logic elements and 85% of available internal memory. Everything fitted and this means the elementary base and structure of the project were selected right.
The circuit helped to save two 50 MHz generators as the project takes only 33 MHz PCI time signal, they are not used and may be not installed on the board in the future. The project works with any of the mother boards that may be found today.
The unit may register all the cards in the network simultaneously provided there are enough GSM channels.

 

Now, in addition to updating the firmware and drivers at out customers, some units are working in our facilities. Some send SMSs for SprintSMS project. The others are "guarded" by our technical support and serve our customers. This helped us understand the essence of our customers' work and what they really need, from the point of view of both ergonomics and additional software functionality. In addition, the unit that works good (not jinxing it!) helped to increase the quality of the software debugging.

SimBank or 200 SmartCards in a box

But this is another story.

 

My thanks to everybody.
To the readers for their interest to the article. To the authors of the other articles for useful lessons. To Habr creators for Habr. To my colleagues for their help and experience they shared with me. To my manager for understanding.

 

P.S. Bonus: SimBank visual control web interface picture.
Special thanks for it to Maxim.

SimBank or 200 SmartCards in a box 

Here, SimBank is launched in 100 cards mode.

SimBank or 200 SmartCards in a box

The list of cards with the information read from them. For the cards, that require PIN code, some information is unavailable.

SimBank or 200 SmartCards in a box

 

Web interface is available from a smart phone, too.
The cards that are in operation right now are highlighted yellow.

Quick contact


 

Contact Details

POLYGATOR (manufacturer)

This email address is being protected from spambots. You need JavaScript enabled to view it.
www.polygator.com
Skypepolygator

 

Dealer contacts:

UKRAINE
Elgato Communications
+380(63)797-3020 (mob)
+380(56)719-9030 (tel./fax office)

Local landlines

RUSSIA
+7(499)403-3248 (Moscow)
UK
+44(203)769-1858 (London)
USA
+1(312)3-400-898 (Chicago)

 

Compatibility Short-list

Native drivers Certified with Tested with Technologies
Asterisk
(PCI GSM Interface Card for 4/8 SIM-cards)

Elastix

SoftSwitch, IP-PBX
Asterisk
3CX
Oktell

VoIP Billing
MVTS
A2Billing
MOR

SMS software
Diafaan
Kannel
Ozeki
ActivExperts
SMS.Gate

GSM UMTS PRI(E1) SMS

SIP

Codecs
G.729 G.711 G.723 G.726 DTMF

SMS protocols
SMPP SMTP AT-commands SMS-via-SQLLite (Under Asterisk)

+ Dev by abc7