Visualize nanometric structures in real time

A real-time reconstruction of platinum nanoparticles on a carbon nanowire produced with the weighted back-projection algorithm in tomviz.

Computer chip designers, materials scientists, biologists and other scientists now have an unprecedented level of access to the world of nanoscale materials through 3D visualization software that connects directly to a microscope. electronic. It allows researchers to see and manipulate 3D visualizations of nanomaterials in real time.

Developed by a team of engineers and software developers led by the University of Michigan, the features are included in a new beta version of tomviz, an open-source 3D data visualization tool already used by tens of thousands of researchers . The new version reinvents the visualization process, allowing you to go from microscope samples to 3D visualizations in minutes instead of days.

In addition to generating results faster, the new features allow researchers to view and manipulate 3D visualizations during an ongoing experiment. This could dramatically speed up research in areas such as microprocessors, electric vehicle batteries, lightweight materials and many more.

“It’s been a long-time dream of the semiconductor industry, for example, to be able to do tomography in a day, and here we’ve squeezed it down to under an hour,” Robert said. Hovden, assistant professor of materials science and engineering. at UM and corresponding author of the study published in Nature Communications. “You can start interpreting and doing science even before you’ve completed an experiment.”

A real-time reconstruction of hyperbranched cobalt phosphate nanoparticles produced with the Simultaneous Iterative Reconstruction Technique algorithm in tomviz.
This rendering of platinum nanoparticles on a carbon backing shows how tomviz interprets microscopy data as it is created, going from a shaded image to a detailed rendering.

Hovden explains that the new software extracts data directly from an electron microscope when it is created and displays the results immediately, a fundamental change from previous versions of tomviz. In the past, researchers collected data from the electron microscope, which took hundreds of two-dimensional projection images of a nanomaterial from several different angles.

Then Hovden and his colleagues took the projections back to the lab to interpret and prepare them before sending them to tomviz, which would take several hours to generate a 3D visualization of an object. The whole process took days to a week, and a problem with one step in the process often meant starting over.

The new version of tomviz does all the interpretation and processing in-place. Researchers get a dark but useful 3D rendering in minutes, which gradually turns into a detailed visualization.

“When you’re working in an invisible world like nanomaterials, you never really know what you’re going to find until you start seeing it,” Hovden said. “So the ability to start interpreting and making adjustments while you’re still under the microscope makes a huge difference to the research process.”

The sheer speed of the new process could also be useful in industry – semiconductor chip makers, for example, could use tomography to perform tests on new chip designs, looking for failures in nanoscale circuitry. 3D far too small to see. In the past, the tomography process was too slow to run the hundreds of tests required in a commercial installation, but Hovden believes tomviz could change that.

Hovden points out that tomviz can be run on a standard consumer laptop. It can connect to newer or older models of electron microscopes. And because it’s open-source, the software itself is available to anyone.

“Open source software is a great tool for empowering science globally. We have made the connection between tomviz and the microscope independent of the microscope manufacturer,” he said. “And because the software doesn’t only looks at the microscope data, he doesn’t care whether this microscope is UM’s latest model or a 20-year-old machine.”

This diagram illustrates the process of extracting two-dimensional projection images from an electron microscope and rendering them into a three-dimensional visualization.
This diagram illustrates the process of extracting two-dimensional projection images from an electron microscope and rendering them into a three-dimensional visualization.

To develop the new capabilities, the UM team drew on its longstanding partnership with software developer Kitware and also brought in a team of scientists who work at the intersection of data science, materials science and microscopy. Early in the process, Hovden worked with Marcus Hanwell of Kitware and Brookhaven National Laboratory to refine the idea of ​​a version of tomviz that would allow real-time visualization and experimentation.

Next, developers from Hovden and Kitware collaborated with UM materials science and engineering graduate researcher Jonathan Schwartz, microscopy researcher Yi Jiang, and machine learning and materials science expert Huihuo Zheng, both from Argonne National Laboratory, to create algorithms capable of quickly and accurately transforming electron microscopy images. in 3D visualizations.

Once the algorithms were completed, David Muller, a professor of applied physics and engineering at Cornell University, and Peter Ericus, a researcher at the Molecular Foundry at Berkeley Lab, worked with Hovden to design a user interface that would support the new abilities.

Finally, Hovden teamed up with materials science and engineering professor Nicholas Kotov, undergraduate scientist Jacob Pietryga, biointerfaces researcher Anastasiia Visheratina, and chemical engineering researcher Prashant Kumar, all at UM, to synthesize a nanoparticle that could be used for real-world testing. new abilities, both to ensure their accuracy and show off their abilities.

They opted for a helix-shaped nanoparticle, about 100 nanometers wide and 500 nanometers long. The new version of tomviz worked as expected; within minutes, it generated a dark image but detailed enough for researchers to make out key details like how the nanoparticle twists, known as chirality. About 30 minutes later, the shadows resolved into a detailed three-dimensional visualization.

A screenshot of tomviz 2.0
A screenshot of tomviz 2.0

The source code for the new tomviz beta is available for free download on GitHub. Hovden believes this will open up new possibilities in areas beyond materials-related research; fields like biology are also set to benefit from access to real-time electron tomography. He also hopes that the project’s “software as science” approach will spur new innovations in science and software development.

“We really have an interdisciplinary approach to research at the intersections of computer science, materials science, physics and chemistry,” Hovden said. “It’s one thing to create really cool algorithms that only you and your graduate students know how to use. It’s another thing if you can enable labs around the world to do these state-of-the-art things.

Kitware collaborators on the project were Chris Harris, Brianna Major, Patrick Avery, Utkarsh Ayachit, Berk Geveci, Alessandro Genova, and Hanwell. Kotov is also Irving Langmuir University Emeritus Professor of Chemical Science and Engineering, Joseph B. and Florence V. Cejka Professor of Engineering, and Professor of Chemical Engineering and Macromolecular Science and Engineering.
“I’m excited for all the new scientific discoveries and 3D visualizations that will come out of the materials science and microscopy community with our new real-time tomography framework,” Schwartz said.

Comments are closed.