Storage Commitment

What is DICOM Storage Commitment Service and why is it needed

Storage commitment (SCM) is a DICOM service that lets you verify if files that were previously sent to the PACS using the DICOM Storage Service were indeed stored by the application you sent it to. The SOP Class UID of this Service is “1.2.840.10008.1.20.1”. One can argue if it is necessary or not because when you send a DICOM image using C-STORE command and get a success status (0x0000) then it is supposed to be stored so the existence of Storage Commitment raises doubts about the meaning of that status in the first place. However, I can defiantly think of reasons for having such service, first because better safe than sorry and second because I already had some programming experience in the days when the DICOM standard was specified, Thanks god, we did make a long way since then. For example, some engineers, for the sake of efficiency and performance considerations, may have decided to first puts the files in a temporary storage or a queue, without even looking at their content and reply immediately with success and then later, when some batch or another thread processes the files in the queue and try to fill the database errors occur. I wouldn’t implement it this way, and I’ll give you reasons for that at the end of this post, but I did run into such implementations. The DICOM standard gives us the service but doesn’t go into the details of what is the implementation meaning should be but IHE does. IHE says that if your application creates instances and send it to somewhere, before deleting them from your local disk it should send a Storage Commitment and if all instances are OK, go ahead and make some space on your hard drive. Sounds like a good idea to me, it’s like double booking. Storage is the transaction and Storage Commitment is the reconciliation, Why not.

Storage Commitment Data Flow

All together SCM is pretty straight forward. All we need to do is to send a list of the instances and get back a reply saying which are in the PACS database and which are not, and that’s exactly how it works. Well, almost.  


The above diagram that I made hopefully explains it all. On the left there’s the SCM request. It is sent using N-ACTION command with a dataset that contains:
  1. A Transaction UID, identifying this commit request
  2. A referenced SOP Sequence, DICOM Tag (0008,1199) with a list of SOP Class UID’s and SOP Instance UID’s we request to commit.
During the N-ACTION all that the SCP does is to get the dataset and only say “I got it”. BTW, we need a SOP Instance UID for sending the N-ACTION because there’s an affected SOP Instance UID there (Why can’t this serve as the transaction UID? Don’t know, maybe you can answer that) so there’s a ‘well known UID 1.2.840.10008.1.20.1.1 and if you use RZDCX you shouldn’t be worried about it or about the SOP Class UID as well. On the right side of the diagram we have the result that is sent independently after the request has been received and processed. The SCP sends the Storage Commit result using N-EVENT-REPORT command that contain:
  1. The transaction UID from the request.
  2. A list of succeeded instances
  3. If not all were ok, a list of failed instances
If the failed instances list is not empty, the storage commit result is failed. Otherwise it is succeeded. The N-EVENT-REPORT command has an Event Type ID attribute that in the case of SCM should be 1 if all instances were committed successfully or 2 if some instances failed to commit.

The Timing of the Storage Commitment Result

Getting the SCM result can sometimes be tricky. Maybe a similar design paradigm to the one described earlier led to this. Maybe, the SCP can’t answer immediately and it needs to think about it, queue the request for some batch process, check the database, compose a reply, queue it for sending and so on. Instead of just getting the results immediately in a response, DICOM lets the SCM SCP the freedom to decide when and how to answer. The SCP should send us back a N-EVENT REPORT command with the result and this result can arrive in one of three ways:
  1. The SCP can Send the N-EVENT-REPORT on the same association that the SCU initiated or
  2. The SCP can start another association to the SCU and send the N-EVENT-REPORT, or
  3. The next time the SCU starts an association the SCP can send the N-EVENT-REPORT immediately after the association negotiation phase.
The 3rd option can actually kill many DICOM Clients that implement Storage Commitment. Many implementations just don’t expect to get a command from the called AE just when they are about to send something over. This option was added to address the following scenario that allegedly can happen with mobile modalities like Cardiac Echo and are hooked in and out from the network plugs at the patient bedside. The explanation that DICOM gives for this behavior is that if for example a cardiac Doppler on a bed-side wheel is hooking in and out of the network, it might hook out before the PACS was able to initiate an association to send the result so the next time it calls in, the PACS  send the result. Cleaver, we knock on the PACS door (port) and the PACS opens up and says: ‘Oh, about your last commit request, here’s the result’. Another thing, options 2 and 3 implies that the PACS should start an association with us so we have to be fully pre-configured there with AE Title, IP Address and port number.

Implementing DICOM Storage Commit SCU with RZDCX

Because the result may be coming on another association, we better have an accepter running. We don’t have to but it’s a good idea because some SCP’s will not do it any other way. In DCXACC there’s a callback named OnCommitResult that hands out the Transaction UID’s and the succeeded and failed instances lists. We can run this accepter on a different process, on a different thread or on the same thread as you’ll see in the example. To send the request you can either call CommitFiles or CommitInstances. If you call the first, RZDCX will open each file, extract the SOP Class UID and SOP Instance UID and build the request dataset. If you use CommitInstances than you have to provide the list. CommitFiles is handier though because you’re not going to delete these files before you got the commit result anyhow. These two methods will not wait for the result and hang out immediately. There’s also CommitFilesAndWaitForResult and its pair CommitInstancesAndWaitForResult that wait for a while before hanging out and gives you the same out parameters as OnCommitResult does. If your PACS support that, these would be easier. Decent PACS should have a flag that controls this behavior for every AE Title and let you select between the ways that the results are sent back to the SCU. Here’s a single threaded example. What is done here is to set an accepter and start it, then send the request, then wait for the result on the accepter. I don’t recommend doing it this way but it kind of cool as an example. The best way I think is to have the accepter run independently on another thread or process that if you implement Storage SCP (which you probably do in order to get instances back on Q/R) also handles incoming C-STORE’s.

C# Test Code

The example code this time is directly from my nUnit test suite.  You can download the sources from this link. All these examples do basically the same: Create some DICOM Test Files, Save them to disk, Send them using C-STORE, Check that they were stored with Storage Commitment and wait for the result.


        public void CommitFilesSameThread()
        {
            // Create test files
            String fullpath = "SCMTEST";
            Directory.CreateDirectory(fullpath);
            CommonTestUtilities.CreateDummyImages(fullpath, 1, 1);

            // Send test files
            string MyAETitle = System.Environment.GetEnvironmentVariable("COMPUTERNAME");
            DCXREQ r = new DCXREQ();
            string succeededInstances;
            string failedInstances;
            r.Send(MyAETitle, IS_AE, IS_Host, IS_port, fullpath + "\\SER1\\IMG1", out succeededInstances, out failedInstances);
            Assert.That(failedInstances.Length == 0);
            Assert.That(succeededInstances.Length > 0);

            // Commit files and wait for result on separate association for 30 seconds
            SyncAccepter a1 = new SyncAccepter();
            r.CommitFiles(MyAETitle, IS_AE, IS_Host, IS_port, fullpath + "\\SER1\\IMG1");
            a1.WaitForIt(30);

            if (a1._gotIt)
            {
                // Check the result

                Assert.True(a1._status, "Commit result is not success");
                Assert.That(a1._failed_instances.Length == 0);

                DCXOBJ obj = new DCXOBJ();
                obj.openFile(fullpath + "\\SER1\\IMG1");
                string sop_class_uid = obj.getElementByTag((int)DICOM_TAGS_ENUM.sopClassUid).Value.ToString();
                string instance_uid = obj.getElementByTag((int)DICOM_TAGS_ENUM.sopInstanceUID).Value.ToString();
                Assert.AreEqual(a1._succeeded_instances, sop_class_uid + ";" + instance_uid + ";");
            }
            else
                Assert.Fail("Didn't get commit result");

            /// Cleanup
            Directory.Delete(fullpath, true);
        }

Here’s the sync accepter:
        class SyncAccepter
        {
            public bool _gotIt = false;
            public bool _status = false;
            public string _transaction_uid;
            public string _succeeded_instances;
            public string _failed_instances;
            public DCXACC accepter;
            public string MyAETitle;

            public void accepter_OnCommitResult(
                bool status,
                string transaction_uid,
                string succeeded_instances,
                string failed_instances)
            {
                _gotIt = true;
                _status = status;
                _transaction_uid = transaction_uid;
                _succeeded_instances = succeeded_instances;
                _failed_instances = failed_instances;
            }

            public SyncAccepter()
            {
                accepter = new DCXACC();
                accepter.OnCommitResult += new IDCXACCEvents_OnCommitResultEventHandler(accepter_OnCommitResult);
                MyAETitle = System.Environment.GetEnvironmentVariable("COMPUTERNAME");
                accepter.WaitForConnection(MyAETitle, 104, 0);
            }

            public bool WaitForIt(int timeout)
            {
                if (accepter.WaitForConnection(MyAETitle, 104, timeout))
                    return accepter.WaitForCommand(timeout);
                else
                    return false;
            }
        }

And here’s the example that waits for the results on the same association.
        public void CommitFilesAndWaitForResultOnSameAssoc()
        {
            bool status = false;
            bool gotIt = false;
            String fullpath = "SCMTEST";
            Directory.CreateDirectory(fullpath);
            CommonTestUtilities.CreateDummyImages(fullpath, 1, 1);
            string succeededInstances;
            string failedInstances;
            string MyAETitle = System.Environment.GetEnvironmentVariable("COMPUTERNAME");
            DCXREQ r = new DCXREQ();
            r.OnFileSent += new IDCXREQEvents_OnFileSentEventHandler(OnFileSent);
            r.Send(MyAETitle, IS_AE, IS_Host, IS_port, fullpath + "\\SER1\\IMG1", out succeededInstances, out failedInstances);
            DCXOBJ obj = new DCXOBJ();
            obj.openFile(fullpath + "\\SER1\\IMG1");
            string sop_class_uid = obj.getElementByTag((int)DICOM_TAGS_ENUM.sopClassUid).Value.ToString();
            string instance_uid = obj.getElementByTag((int)DICOM_TAGS_ENUM.sopInstanceUID).Value.ToString();
            string transactionUID = r.CommitFilesAndWaitForResult(MyAETitle, IS_AE, IS_Host, IS_port, fullpath + "\\SER1\\IMG1",
                5, out gotIt, out status, out succeededInstances, out failedInstances);
            Directory.Delete(fullpath, true);

            Assert.True(status, "Commit result is not success");
            Assert.That(failedInstances.Length == 0);
            Assert.AreEqual(succeededInstances, sop_class_uid + ";" + instance_uid + ";");
        }

And here’s the common test utilities class in case you need it
using System;
using System.Collections.Generic;
using System.Text;
using rzdcxLib;
using System.IO;

namespace rzdcxNUnit
{
    class CommonTestUtilities
    {
        public static string TestPatientName
        {
            get { return "John^Doe"; }
        }

        public static string TestPatientID
        {
            get { return "123765"; }
        }

        public static string TestStudyInstanceUID
        {
            get { return "123765.1"; }
        }

        public static string TestSeriesInstanceUID
        {
            get { return "123765.1.1"; }
        }

        ///






        /// Create to series with 4 images each of test images
        ///

        /// Root directory to put the files in
        /// a list of the filenames of the created images
        public static unsafe List<String> CreateDummyImages(String path)
        {
            return CreateDummyImages(path, 4, 2, "", false);
        }
        public static unsafe List<String> CreateDummyImages(String path, int numSeries, int numImagesPerSeries)
        {
            return CreateDummyImages(path, numSeries, numImagesPerSeries, "", false);
        }

        public static unsafe List<String> CreateDummyImages(String path, int numSeries, int numImagesPerSeries, String suffix)
        {
            return CreateDummyImages(path, numSeries, numImagesPerSeries, suffix, false);
        }

        ///






        /// Create a set of test images
        ///

        /// Root directory to put the files in
        /// How many series to create
        /// How many image files per series
        /// filename suffix to use
        /// a list of the filenames of the created images
        public static unsafe List<String> CreateDummyImages(String path, int numSeries, int numImagesPerSeries, String suffix, bool long_uid_names)
        {
            List<String> filesList = new List<string>();
            const int ROWS = 64;
            const int COLUMNS = 64;
            const int SAMPLES_PER_PIXEL = 1;
            const string PHOTOMETRIC_INTERPRETATION = "MONOCHROME2";
            const int BITS_ALLOCATED = 16;
            const int BITS_STORED = 12;
            const int RESCALE_INTERCEPT = 0;

            DCXOBJ obj = new DCXOBJ();

            /// Create an element pointer to place in the object for every tag
            DCXELM el = new DCXELM();

            /// Set Hebrew Character Set
            el.Init((int)DICOM_TAGS_ENUM.SpecificCharacterSet);
            el.Value = "ISO_IR 192";
            /// insert the element to the object
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.Rows);
            el.Value = ROWS;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.Columns);
            el.Value = COLUMNS;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.SamplesPerPixel);
            el.Value = SAMPLES_PER_PIXEL;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.PhotometricInterpretation);
            el.Value = PHOTOMETRIC_INTERPRETATION;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.BitsAllocated);
            el.Value = BITS_ALLOCATED;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.BitsStored);
            el.Value = BITS_STORED;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.HighBit);
            el.Value = BITS_STORED - 1;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.PixelRepresentation);
            el.Value = 0;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.WindowCenter);
            el.Value = (int)(1 << (BITS_STORED - 1));
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.WindowWidth);
            el.Value = (int)(1 << BITS_STORED);
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.RescaleIntercept);
            el.Value = (short)RESCALE_INTERCEPT;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.RescaleSlope);
            el.Value = 1;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.GraphicData);
            el.Value = "456\\8934\\39843\\223\\332\\231\\100\\200\\300\\400";
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.PixelData);
            el.Length = ROWS * COLUMNS * SAMPLES_PER_PIXEL;
            el.ValueRepresentation = VR_CODE.VR_CODE_OW;

            ushort[] pixels = new ushort[ROWS * COLUMNS];
            for (int y = 0; y < ROWS; y++)
            {
                for (int x = 0; x < COLUMNS; x++)
                {
                    int i = x + COLUMNS * y;
                    pixels[i] = (ushort)(((i) % (1 << BITS_STORED)) - RESCALE_INTERCEPT);
                }
            }
            fixed (ushort* p = pixels)
            {
                UIntPtr p1 = (UIntPtr)p;
                el.Value = p1;
            }

            obj.insertElement(el);

            // Set identifying elements
            el.Init((int)DICOM_TAGS_ENUM.PatientsName);
            el.Value = CommonTestUtilities.TestPatientName;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.patientID);
            el.Value = TestPatientID;
            obj.insertElement(el);

            String study_uid = "123765.1";
            el.Init((int)DICOM_TAGS_ENUM.studyInstanceUID);
            el.Value = CommonTestUtilities.TestStudyInstanceUID;
            obj.insertElement(el);

            el.Init((int)DICOM_TAGS_ENUM.StudyID);
            el.Value = 1;
            obj.insertElement(el);


            el.Init((int)DICOM_TAGS_ENUM.sopClassUid);
            el.Value = "1.2.840.10008.5.1.4.1.1.7"; // Secondary Capture
            obj.insertElement(el);


            for (int seriesid = 1; seriesid <= numSeries; seriesid++)
            {

                String series_uid = study_uid + "." + seriesid;
                el.Init((int)DICOM_TAGS_ENUM.seriesInstanceUID);
                el.Value = series_uid;
                obj.insertElement(el);

                string series_path = "";
                if (long_uid_names)
                    series_path = path + "\\" + series_uid;
                else
                    series_path = path + "\\SER" + seriesid;
                Directory.CreateDirectory(series_path);

                el.Init((int)DICOM_TAGS_ENUM.SeriesNumber);
                el.Value = seriesid;
                obj.insertElement(el);

                for (int instanceid = 1; instanceid <= numImagesPerSeries; instanceid++)
                {
                    String instance_uid = series_uid + "." + instanceid;
                    el.Init((int)DICOM_TAGS_ENUM.sopInstanceUID);
                    el.Value = instance_uid;
                    obj.insertElement(el);

                    el.Init((int)DICOM_TAGS_ENUM.InstanceNumber);
                    el.Value = instanceid;
                    obj.insertElement(el);

                    /// Save it
                    ///
                    String filename = "";
                    if(long_uid_names)
                        filename = series_path + "\\" + instance_uid + suffix;
                    else
                        filename = series_path + "\\IMG" + instanceid + suffix;
                    obj.saveFile(filename);
                    filesList.Add(filename);
                }
            }
            return filesList;
        }

        // The unsafe keyword allows pointers to be used within the following method:
        public static unsafe void Copy(byte* pSrc, int srcIndex, byte[] dst, int dstIndex, int count)
        {
            if (pSrc == null || srcIndex < 0 ||
                dst == null || dstIndex < 0 || count < 0)
            {
                throw new System.ArgumentException();
            }

            //int srcLen = src.Length;
            //int dstLen = dst.Length;
            //if (srcLen - srcIndex < count || dstLen - dstIndex < count)
            //{
            //    throw new System.ArgumentException();
            //}

            // The following fixed statement pins the location of the src and dst objects
            // in memory so that they will not be moved by garbage collection.
            fixed (byte*  pDst = dst)
            {
                byte* ps = pSrc;
                byte* pd = pDst;

                // Loop over the count in blocks of 4 bytes, copying an integer (4 bytes) at a time:
                for (int i = 0; i < count / 4; i++)
                {
                    *((int*)pd) = *((int*)ps);
                    pd += 4;
                    ps += 4;
                }

                // Complete the copy by moving any bytes that weren't moved in blocks of 4:
                for (int i = 0; i < count % 4; i++)
                {
                    *pd = *ps;
                    pd++;
                    ps++;
                }
            }
        }
    }
}

Summary

Let’s wrap it all up:
  1. There’s DICOM service called Storage Commitment (SCM)  that gets a list of instances and gives back which are stored with our peer and can be safely deleted from our local disk and which are not and we better send them over again.
  2. The SOP Class UID of Storage Commitment is 1.2.840.10008.1.20.1
  3. The request is sent via N-ACTION with the well known SOP Instance UID
  4. The result is received via N-EVENT-REPORT
  5. The result can come on a separate association.
  6. If we want to give the PACS a chance to send the result on the same association, we should at least wait for it to come for couple of seconds after sending the request.

Q&A

Q: Should the list of instances in the commit request be identical to the group of instances we sent in the association that stored the files?
A: No. In DICOM there's no contextual meaning to the association.
Q: If some files failed to commit, should I send all files again or just the ones that failed?
A:I would recommend sending just the one that failed.
Q: What if I didn't get the result?
A: Send a Storage Commit Request again.
Q: How can I know the reason for the failure from the commit result?
A: You can't but from the C-STORE command response you can. There's a status there and sometimes additional explanation attributes. Read the log. 

Why not to implement it the way described above

One last thing I owe you. You can say that the command succeeded but the file is dead, ha? After all, we are in a hospital right, the operation was successful but the patient died? anyone? Never mind. Implementations that just store the file on disk, say OK and later parse it and find out that it’s wrong and can’t actually process it simply don’t have any way to tell the client what was the problem with the file. They lose the only chance to respond properly in the C-STORE Response. If you fear that processing the registration of a new instance during the C-STORE will delay the communication so you better review your registration process. Maybe your DICOM parser is too slow or your database connection is not optimized.

<- Prev - Modality Performed Procedure Step                                                        

Pixel Data - Next ->

6 comments:

  1. Excellent article.
    Thank you.
    Ariel T.

    ReplyDelete
  2. Hello,
    Thanks for this great tutorial. I would like to hear your opinion about the following case:
    I am implementing a DICOM router, it receives a full study very fast in a local device, and then uploads it to a repository on the cloud.
    So, I am wondering, when is the best moment to confirm the arrival of the image/s? at router or cloud repository? What to answer when the images are still in the router buffer?
    Thanks in advance,
    Jaime O.

    ReplyDelete
    Replies
    1. Hi Jamie
      Thats an interesting question.
      I would suggest to check what IHE has to say about it. The key to search would be 'shared image archive' in the XDS-I integration profile.
      I'll do this check as well and re-reply.
      The way I would do it is to reply success once it's stored on the proxy regardless of the upload progress.
      Otherwise, if you reply failed, the modality will send the study again and this is not what you want.
      Regards
      Roni

      Delete
  3. Excellent Article!

    Thanks! Now I have a idea on how the whole SC works!

    Cheers,
    Navin.

    ReplyDelete
  4. Excellent Articles.

    Hey,

    I am using DCM4CHEE as my PACS. It supports the Storage Commit but I am not able to figured out how the SCP can start another association to the SCU and send the N-EVENT-REPORT using DCM4CHEE. do you have any idea. by default, it support two way communication, means send the N-Action Report using new association. Do you have any idea hwo to enable one way communication?

    Thanks,
    Deepak

    ReplyDelete
    Replies
    1. Thanks. Unfortunately I'm not a Java guy. Most of our work is in C# and C++. I've used DCM4CHEE maybe once or twice so can't really help. Sorry.

      Delete