Blog Post Icon
Blog
05/30/2016

How to export and import field collection data in Drupal

Hello!

There are many great tools out there that make importing and exporting content into Drupal nodes very easy. We deal with custom Drupal content often and have extensively used many of the (amazing) tools available, such as node export, views data export and feed import , specifically with the field collection feeds addon.

With all of these solutions, we had limited but no success in exporting our custom content + field collection data. Our field collections contained fields that were configured to store multiple values as an array, which may be why. We never tried the tools that didn’t work with different variations of field options in our field collections. They just didn’t work for us.

So we looked into how hard it would be to write our own drush command to easily integrate into our code push system (the system through which code and data is propagated from our staging to production environments). All we needed was a simple way to export Drupal content to a CSV and then import from the CSV back into Drupal whenever needed.

We thought maybe we could save other people from going through (most) of the trouble that we had to go through to put this together by sharing our process and the code we used to create the drush command. Ultimately we could release this as a drupal module for a wider audience if there’s enough demand for it.

Export data to a CSV

The easier part of this process involves pulling all the node data and exporting it as a CSV for later use. Putting a CSV together in PHP is relatively straightforward, however I’ll paste the function for exporting below as a reference and explain further down :

function star_export() {

        $fc_name = drush_get_option('c');

        // we will use SPL to send our data to STDOUT formatted as CSV
        $fout = new \SplFileObject("php://stdout");
        // write our headers
        $fout->fputcsv([
                'nid', 'title', 'path_alias', 'field_collection_name', 'assigned_nodes'
        ]);
        // use a generator to loop through nodes
        foreach (nodes_generator($fc_name) as $node) {
                $fout->fputcsv([
                // nid
                $node[0]->nid,
                // title
                $node[0]->title,
                // path alias
                url('node/' . $node[0]->nid),
                // field collection name
                $fc_name,
                // assigned nodes
                $node[1]
                ]);
        }
}

This is the function triggered by an “Export” command within drush. You can see we’re looking for an option passed called “c”. The syntax would be –c=”field_collection_name”. Because this module was written for our own use, we didn’t put much emphasis on implementing a dynamic command that can simply read all field collections for a content type and then parse all the fields, though if someone wanted to branch our code on github (link at bottom) and contribute that kind of dynamic functionality, that would be most helpful.

You can also see that we have a simple format for our CSV :

        $fout->fputcsv([
                'nid', 'title', 'path_alias', 'field_collection_name', 'assigned_nodes'
        ]);

Everything’s pretty self explanatory, right? Well you’ll notice in the code snippet that I’m calling a function called node_generator. This is where most of the magic happens when exporting the data.

function nodes_generator($fc_name) {
        static $count; // prevent infinite loops
        // query for nodes newer than the specified date
        $query = $query = new EntityFieldQuery();
        $query->entityCondition('entity_type', 'node')
                ->propertyCondition('type', 'custom_content')
                ->propertyCondition('status', 1);
        $result = $query->execute();
        if (!empty($result)) {
                foreach ($result['node'] as $nid => $row) {
                        $count++;
                        // TRUE will reset static cache to keep memory usage low
                        $node = node_load($row->nid, null, TRUE);
                        $collections = array();
                        $field_collection_fields = field_get_items('node', $node, $fc_name);
                        $collection_item_final = array();

                        if (!empty($field_collection_fields)) {
                                $cnt = 0;
                                foreach ($field_collection_fields as $field_collection_field) {
                                        $item = field_collection_field_get_entity($field_collection_field);
                                        $collection_item = $item->field_first_option['und'][0]['value'] . ',' .  $item->field_second_option['und'][0]['value'];
                                        (count($field_collection_fields) >= 2) ? $field_divider = '^' : $field_divider = NULL;
                                        for ($i = 0;$i < count($item->field_grp_assigned);$i++) {
                                                $collection_items = array();
                                                for ($j = 0;$j < count($item->field_grp_assigned['und']);$j++) {
                                                        $collection_items[] = $item->field_grp_assigned['und'][$j]['target_id'];
                                                }
                                                // dont add divider if last iteration of loop
                                                if ($cnt == count($field_collection_fields) - 1) {
                                                        $collection_item_final[] = $collection_item . ',' . implode('|', $collection_items);
                                                } else {
                                                        $collection_item_final[] = $collection_item . ',' . implode('|', $collection_items) . $field_divider;
                                                }

                                        }
                                        $cnt++;
                                }
                                // return assigned collection
                                yield array($node, implode($collection_item_final));
                        }  else {
                                // return node without assignments
                                yield array($node);
                        }
                }
        }
        return;
}

The bulk of the work done in the above nodes_generator function is pulling the fields from the provided field collection (passed from the export function that triggers the nodes_generator function) and dealing with the data by way of a few nested for-loops.

Again this function is not very dynamic and either would have to be customized to accommodate your custom content or perhaps customized so that you could use additional argument and option fields in the drush command to customize the output. When run, you will see a CSV file get generated with output that looks similar to the following :

nid,title,path_alias,field_collection_name,assigned_nodes
0001,"node title 1",/en/node/0001,,,
0002,"node title 2",/en/node/0002,"field_collection_grp1",,"option 1","option 2",0003|0004|0005|0006"

The “assigned_nodes” field is the field collection field data I’m pulling and exporting. The “field_collection_name” is self explanatory. The fields “option 1” and “option 2” are fields in the collection group that only contain 1 value (not multi value as with the last field).

The last field in the CSV is assigned NIDs from an Entity reference views widget that allows the user to search for nodes and “Assign” them to the field. All I want to export for this is the node IDs that serve as the “target_id” reference in the backend. You can see each field in the CSV is separated by a comma. In the last field, multi fields are further separated by the pipe character for parsing during import.

Import data from the CSV

The export process again is much simpler than the import process. For the import process, we need to specify a source file and then we need to loop through all the CSV items in order to prepare the data to create a new node and save the data. All the data creation and manipulation is done using Drupal’s Entity metadata wrappers.

function star_importer() {

        $file = drush_get_option('f');
        if (!empty($file) && file_exists($file)) {
                $csv_data = array_map('str_getcsv', file($file));
                // loop through array and skip first line
                for ($i = 1; $i < count($csv_data);$i++) {
                        $nid = $csv_data[$i][0];
                        $title = $csv_data[$i][1];
                        $path_alias = $csv_data[$i][2];
                        $fc_name = $csv_data[$i][3];
                        $fc = $csv_data[$i][4];

                        // fail if key fields are not present
                        if (empty($nid) || empty($title)) {
                                return 'Key fields are missing from array '.$i.'. Check the file and try again';
                        }

                        // create a new node and assign the data
                        $values = array(
                                'type' => 'custom_content',
                                'uid' => 1,
                                'status' => 0,
                                );
                        $entity = entity_create('node', $values);
                        $ewrapper = entity_metadata_wrapper('node', $entity);
                        $ewrapper->title->set($title);
                        // if there's field collection defined
                        if (!empty($fc)) {
                                $fc_items = explode('^', $fc);
                                foreach ($fc_items as $fc_item) {
                                        $fc_array = explode(',', $fc_item);
                                        $first_opt = $fc_array[0];
                                        $second_amt = $fc_array[1];
                                        $fc_nodes = explode('|', $fc_array[2]);
                                        if (empty($first_opt) || empty($second_amt)) {
                                                echo 'This assignment is missing either the first or second field option, skipping ..';
                                                continue;
                                        }
                                        $collection = entity_create('field_collection_item', array('field_name' => $fc_name));
                                        $collection->setHostEntity('node', $entity);
                                        $cwrapper = entity_metadata_wrapper('field_collection_item', $collection);
                                        $cwrapper->field_first_opt->set($first_opt);
                                        $cwrapper->field_second_opt->set($second_opt);
                                        $cwrapper->field_grp_assigned->set($assigned_nodes);
                                        $cwrapper->save();
                                }
                        }

                        // save
                        $ewrapper->save();
                }
        } else {
                return 'No file given to import or file does not exist';
        }
}

There’s quite a lot going on in the above function! Sometimes nested for-loops can make one’s head spin. However its the only way to deal with the multi dimensional arrays as well as the multitude of arrays themselves. There could be much more sanity checks and whatnot, but again remember this is just proof of concept.

The main thing that should be discussed here within the context of the above function is the field collection portion of the function. Basically what took a bit of troubleshooting was how to take the data saved from the CSV and then create a field collection reference for the newly created node and save all the field collection data, including the multi-valued field_grp_assigned collection field.

Remember that CSV field that is separated by the pipe character? Well we use the explode php function to convert it into a proper array and then use the entity wrapper set function to assign the array to the field. The other field sets are either static (single value) variables in the collection or within the regular node fields themselves.

This took some time to put together so I hope its of use to someone out there! Feel free to check out the github repo for the above code. Contribute or branch it into a functional + dynamic module! šŸ™‚

GitHub Repository

At Shift8, we cater to all sorts of businesses in and around Toronto from small, medium, large and enterprise projects. We are comfortable adapting to your existing processes and try our best to compliment communication and collaboration to the point where every step of the way is as efficient as possible.

Our projects are typically broken into 5 or 6 key “milestones” which focus heavily on the design collaboration in the early stages. We mock-up any interactive or unique page within your new website so that you get a clear picture of exactly how your design vision will be translated into a functional website.

Using tools like Basecamp and Redpen, we try to make the process simple yet fun and effective. We will revise your vision as many times as necessary until you are 100% happy, before moving to the functional, content integration and development phases of the project.

For the projects that are more development heavy, we make sure a considerable amount of effort is spent in the preliminary stages of project planning. We strongly believe that full transparency with a project development plan ensures that expectations are met on both sides between us and the client. We want to ensure that the project is broken into intelligent phases with accurate budgetary and timeline breakdowns.

Approved design mock-ups get translated into a browse-ready project site where we revise again and again until you are satisfied. Client satisfaction is our lifeblood and main motivation. We aren’t happy until you are.

Need Web Design?

Fill out the form to get a free consultation.

shift8 web toronto – 416-479-0685
203A-116 geary ave. toronto, on M6H 4H1, Canada
Ā© 2023. All Rights Reserved by Star Dot Hosting Inc.

contact us
phone: 416-479-0685
toll free: 1-866-932-9083 (press 1)
email: sales@shift8web.com

Shift8 Logo