View topic - distributed rendering shell script for openMosix and Yafray

distributed rendering shell script for openMosix and Yafray

General questions related with the YafaRay Project, 3D computer graphics and about this site.

distributed rendering shell script for openMosix and Yafray

Post Thu Jan 13, 2005 8:34 am

Hello, all. I wanted to see if I could make a distributed rendering program for yafray using just the command line interface. This script is definitely not the best, but it works. I've been able to reduce my rendering times by greater than half with my puny cluster (see below).

There are some requirements.
1. You need to be running openMosix. I'm just using good old clusterknoppix on 4 nodes. One is an athlon xp 3200+ with 2 GB RAM, three others are little crappy Dell optiplexes with 128 MB RAM and pentiumIII processors. Not much of a cluster, but hey, it works.
2. You need to have passwordless SSH access to all your nodes from the node you start the script from. Read the SSH man if you need help setting this up or search the web.
3. You need to have the mosix file system (MFS) set up. I have mine set to /mfs. If you don't know what this is, read more about it from the openMosix website.
4. You need the program xmlstarlet which is handy command line XML toolkit.
5. You need imagemagick.
6. Finally, all paths have to be absolute with respect to the MFS. So, in your XML, all images, textures, outfile paths have to be something like /mfs/node/path/to/file.
7. Command line useage:
/bin/bash distributedrender.sh /mfs/node/path/to/XMLsource.xml
Example from my workstation (all one line):
/bin/bash /home/renderer/shellscripts/distributedrender.sh /mfs/357/home/renderer/benchmarks/simplecube.xml

I'm hoping this will be useful to some of you. Any suggestions, comments, flames, etc can be directed to mattbatten@gmail.com. Enjoy.

Code: Select all
#!/bin/bash
#
#    Distributed rendering shell script for Yafray and openMosix
#      by
#   Matthew Batten
#   mattbatten@gmail.com
#   Copyright 2004
#   version 0.1
#   released January 10, 2005

STARTTIME=`date +%s`

# Get the file parameter sent from the command line.
XMLSOURCE=$1
#echo "The xml source was $XMLSOURCE."

#Set the number of COLUMNS, will be incremented by counting the number of nodes
COLUMNS=0
# Get the list of all nodes by using the directory
# listing of /proc/hpc/nodes.
directory=/proc/hpc/nodes
listoffiles=`ls $directory | sort` 2>/dev/null
declare -a LISTOFNODES
# For each node, get the IP and add a tick to the COLUMNS counter
for node in $listoffiles    # For each directory representing a node in the cluster
do
  LISTOFNODES[$COLUMNS]=`mosctl whois $node`
  COLUMNS=`expr $COLUMNS + 1`
done
echo "COLUMNS = $COLUMNS"

# Get the x-axis resolution of the output image from the scene/camera/@resx value in the XML
RESX=`/usr/bin/xmlstarlet sel -t -v "scene/camera/@resx" $XMLSOURCE`
echo "RESX = $RESX"

# Get the y-axis resolution of the output image from the scene/camera/@resx value in the XML
RESY=`/usr/bin/xmlstarlet sel -t -v "scene/camera/@resy" $XMLSOURCE`
echo "RESY = $RESY"

# Based on $RESX and $COLUMNS we can calculate the yafray region column width
# In the case where the number of nodes is odd-numbered, we set the range wider and then later we split the last column into two.
if [ `expr $COLUMNS % 2` = 0 ]  # If we have an even number of nodes...
then   # Just set column ranges as even dividends
   COLUMNRANGE=`echo 2 $COLUMNS | awk '{print $1 / $2}'`         # We use 2 because yafray goes from -1 to 1
   echo "COLUMNRANGE = $COLUMNRANGE"
   COLUMNPIXELRANGE=`echo $RESX $COLUMNS | awk '{print $1 / $2}'`      # Will be needed later for ImageMagick
   echo "COLUMNPIXELRANGE = $COLUMNPIXELRANGE"
else   # Uh-oh, we don't have an even number of nodes, have to be fancy
   #echo "Odd number of nodes, special processing required."
   ODDCOLUMNNUMBER=`expr $COLUMNS - 1`
   ODDCOLUMNRANGE=`echo 2 $ODDCOLUMNNUMBER | awk '{print $1 / $2}'`         # We use 2 because yafray goes from -1 to 1
   echo "ODDCOLUMNRANGE = $ODDCOLUMNRANGE"
   
   ODDCOLUMNPIXELRANGE=`echo $RESX $ODDCOLUMNNUMBER | awk '{print $1 / $2}'`      # Will be needed later for ImageMagick
   echo "ODDCOLUMNPIXELRANGE = $ODDCOLUMNPIXELRANGE"
fi

# Create an array to hold the range tracking values
declare -a RANGETRACKER

# The rangestart is the yafray range start point of -1
rangestart=-1
#echo "rangestart = $rangestart"

# Add to the RANGETRACKER array all the range values that will be used by yafray
if [ `expr $COLUMNS % 2` = 0 ]  # If we have an even number of nodes...
then
   COUNTER=0
   while [ $COUNTER -lt $COLUMNS ]       
   do
      RANGETRACKER[$COUNTER]=$rangestart
      #echo "rangestart = $rangestart"
      #echo "Setting rangestart..."
      rangestart=`echo $rangestart $COLUMNRANGE | awk '{print $1 + $2}'`
      
      echo "RANGETRACKER[$COUNTER] = ${RANGETRACKER[$COUNTER]}"
      
      #echo "Setting counter..."
      COUNTER=`expr $COUNTER + 1`
   done
else   # Uh-oh, we have an odd number of nodes, have to be fancy
   #echo "Odd number of nodes, special processing required to set RANGETRACKER values."
   COUNTER=0
   while [ $COUNTER -lt `expr $COLUMNS - 1` ]    # 2 loops on a 3 node system   
   do
      if [ $COUNTER = `expr $COLUMNS - 2` ]
      then   # Here we are on the last column, which means we will need to split it into two columns
         RANGETRACKER[$COUNTER]=$rangestart
         #echo "rangestart = $rangestart"
         #echo "Setting rangestart..."
         rangestart=`echo $rangestart $ODDCOLUMNRANGE | awk '{print $1 + ($2 / 2)}'`
         echo "RANGETRACKER[$COUNTER] = ${RANGETRACKER[$COUNTER]}"
         
         #echo "Setting counter..."
         COUNTER=`expr $COUNTER + 1`
         
         RANGETRACKER[$COUNTER]=$rangestart
         #echo "rangestart = $rangestart"
         #echo "Setting rangestart..."
         rangestart=`echo $rangestart $ODDCOLUMNRANGE | awk '{print $1 + ($2 / 2)}'`
         echo "RANGETRACKER[$COUNTER] = ${RANGETRACKER[$COUNTER]}"
      else
         RANGETRACKER[$COUNTER]=$rangestart
         #echo "rangestart = $rangestart"
         #echo "Setting rangestart..."
         rangestart=`echo $rangestart $ODDCOLUMNRANGE | awk '{print $1 + $2}'`
         
         echo "RANGETRACKER[$COUNTER] = ${RANGETRACKER[$COUNTER]}"
   
      fi
   
      #echo "Setting counter..."
      COUNTER=`expr $COUNTER + 1`   
   done
fi

# Calculate the pixel buffer width to pad the column sides (prevents the black line issues).
PIXELBUFFERX=`echo $RESX | awk '{print (($1 + 4)/$1)-1}'`
echo "PIXELBUFFERX = $PIXELBUFFERX"
PIXELBUFFERY=`echo $RESY | awk '{print (($1 + 2)/$1)-1}'`
echo "PIXELBUFFERY = $PIXELBUFFERY"

# Calculate the Y range buffer values
YRANGETOP=`echo 1 $PIXELBUFFERY | awk '{print ($1 + $2)}'`
echo "YRANGETOP = $YRANGETOP"
YRANGEBOTTOM=`echo -1 $PIXELBUFFERY | awk '{print ($1 - $2)}'`
echo "YRANGEBOTTOM = $YRANGEBOTTOM"

#Calculate the X range region values (left and right values for each column)
declare -a COLUMNLEFT
declare -a COLUMNRIGHT

# array to hold the render region string to be used by yafray
declare -a RENDERREGION

# Make the render region values for yafray, going from left to right
if [ `expr $COLUMNS % 2` = 0 ]  # If we have an even number of nodes...
then   # Even number of nodes
   COUNTER=0
   while [ $COUNTER -lt $COLUMNS ]
   do
      COLUMNLEFT[$COUNTER]=`echo ${RANGETRACKER[$COUNTER]} $PIXELBUFFERX | awk '{print ($1 - $2)}'`
      COLUMNRIGHT[$COUNTER]=`echo ${RANGETRACKER[$COUNTER]} $COLUMNRANGE $PIXELBUFFERX | awk '{print ($1 + $2 + $3)}'`
      echo "COLUMNLEFT[$COUNTER] = ${COLUMNLEFT[$COUNTER]}"
      echo "COLUMNRIGHT[$COUNTER] = ${COLUMNRIGHT[$COUNTER]}"
      # Push the render region on to the RENDERREGION array
      RENDERREGION[$COUNTER]="${COLUMNLEFT[$COUNTER]}:${COLUMNRIGHT[$COUNTER]}:$YRANGEBOTTOM:$YRANGETOP"
      echo "Region $COUNTER will be ${RENDERREGION[$COUNTER]}"
      
      COUNTER=`expr $COUNTER + 1`
   done
else   # Odd number of nodes
   #echo "Odd number of nodes, special processing required to set COLUMNLEFT and COLUMNRIGHT values."
   COUNTER=0   
   while [ $COUNTER -lt `expr $COLUMNS - 1` ]
   do
      if [ $COUNTER = `expr $COLUMNS - 2` ]
      then   # Here we are on the last column, which means we will need to split it into two columns.
         COLUMNLEFT[$COUNTER]=`echo ${RANGETRACKER[$COUNTER]} $PIXELBUFFERX | awk '{print ($1 - $2)}'`
         COLUMNRIGHT[$COUNTER]=`echo ${RANGETRACKER[$COUNTER]} $ODDCOLUMNRANGE $PIXELBUFFERX | awk '{print ($1 + ($2 / 2) + $3)}'`         
         echo "COLUMNLEFT[$COUNTER] = ${COLUMNLEFT[$COUNTER]}"
         echo "COLUMNRIGHT[$COUNTER] = ${COLUMNRIGHT[$COUNTER]}"
         # Push the render region on to the RENDERREGION array
         RENDERREGION[$COUNTER]="${COLUMNLEFT[$COUNTER]}:${COLUMNRIGHT[$COUNTER]}:$YRANGEBOTTOM:$YRANGETOP"
         echo "Region $COUNTER will be ${RENDERREGION[$COUNTER]}"
         
         COUNTER=`expr $COUNTER + 1`
         
         COLUMNLEFT[$COUNTER]=`echo ${RANGETRACKER[$COUNTER]} $PIXELBUFFERX | awk '{print ($1 - $2)}'`
         COLUMNRIGHT[$COUNTER]=`echo ${RANGETRACKER[$COUNTER]} $ODDCOLUMNRANGE $PIXELBUFFERX | awk '{print ($1 + ($2 / 2) + $3)}'`         
         echo "COLUMNLEFT[$COUNTER] = ${COLUMNLEFT[$COUNTER]}"
         echo "COLUMNRIGHT[$COUNTER] = ${COLUMNRIGHT[$COUNTER]}"
         # Push the render region on to the RENDERREGION array
         RENDERREGION[$COUNTER]="${COLUMNLEFT[$COUNTER]}:${COLUMNRIGHT[$COUNTER]}:$YRANGEBOTTOM:$YRANGETOP"
         echo "Region $COUNTER will be ${RENDERREGION[$COUNTER]}"         
                        
      else
         COLUMNLEFT[$COUNTER]=`echo ${RANGETRACKER[$COUNTER]} $PIXELBUFFERX | awk '{print ($1 - $2)}'`
         COLUMNRIGHT[$COUNTER]=`echo ${RANGETRACKER[$COUNTER]} $ODDCOLUMNRANGE $PIXELBUFFERX | awk '{print ($1 + $2 + $3)}'`
         echo "COLUMNLEFT[$COUNTER] = ${COLUMNLEFT[$COUNTER]}"
         echo "COLUMNRIGHT[$COUNTER] = ${COLUMNRIGHT[$COUNTER]}"
         # Push the render region on to the RENDERREGION array
         RENDERREGION[$COUNTER]="${COLUMNLEFT[$COUNTER]}:${COLUMNRIGHT[$COUNTER]}:$YRANGEBOTTOM:$YRANGETOP"
         echo "Region $COUNTER will be ${RENDERREGION[$COUNTER]}"         
      fi
      
      COUNTER=`expr $COUNTER + 1`   
   done
fi

###############
#  Now that we have all the region information for yafray,
#  all we need to do is change the image output path for each
#  child rendering process. We use xmlstarlet for this.
#
#############
# First, get the original output path and filename
OUTPUTFILE=`/usr/bin/xmlstarlet sel -t -v "scene/render/outfile/@value" $XMLSOURCE`
echo "OUTPUTFILE = $OUTPUTFILE"
# Get the output directory path
#OUTPUTPATH="`expr "$OUTPUTFILE" : '\(.*\)/'`"
#echo "OUTPUTPATH=$OUTPUTPATH"
# Get the file name
#BASENAME="`expr "//$OUTPUTFILE" : '.*/\([^/]*\)'`"
#echo "BASENAME=$BASENAME"
# Get the file suffix
SUFFIX="`expr "$OUTPUTFILE" : '.*\.\([^./]*\)$'`"
echo "SUFFIX=$SUFFIX"
# Get the path + file name
PATHANDFILENAME="`expr "$OUTPUTFILE" : '\(.*\)\.[^./]*$' \| "$XMLSOURCE"`"
echo "PATHANDFILENAME= $PATHANDFILENAME"

# Loop through the number of nodes and add to array that holds file output paths
declare -a OUTPUTCOLUMNPATHS
COUNTER=0
while [ $COUNTER -lt $COLUMNS ]
do
   OUTPUTCOLUMNPATHS[$COUNTER]=$PATHANDFILENAME-$COUNTER.$SUFFIX
   echo "${OUTPUTCOLUMNPATHS[$COUNTER]}"   
   COUNTER=`expr $COUNTER + 1`
done

# Now we can spawn yafray
echo "Starting yafray on $COLUMNS nodes... Please wait until finished"
COUNTER=0
while [ $COUNTER -lt $COLUMNS ]
do
   # 1. Change the output path in the source file XML
   NEWXML=`/usr/bin/xmlstarlet ed -P -u "/scene/render/outfile/@value" -v ${OUTPUTCOLUMNPATHS[$COUNTER]} $XMLSOURCE`
   echo "$NEWXML" > $XMLSOURCE
   # Remove the "<?xml version="1.0"?>" that xmlstarlet automatically adds.
   OLDXML=`cat $XMLSOURCE`
   NEWXML=`echo "$OLDXML" | sed '1d'`
   echo "$NEWXML" > $XMLSOURCE

   # 2. Spawn yafray
   NODE=${LISTOFNODES[$COUNTER]}
   if [ "$NODE" = `hostname -i | sed 's/[ \t]*$//'` ] # The sed action removes the trailing white space from the IP address.
   then
      #echo "yafray -r ${RENDERREGION[$COUNTER]} $XMLSOURCE 1>>$PATHANDFILENAME$COUNTER.log.txt 2>&1 &"
      yafray -r ${RENDERREGION[$COUNTER]} $XMLSOURCE 1>>$PATHANDFILENAME-$COUNTER.log 2>&1 &   # The process is spawned on the local node.
   else
      #echo "ssh $NODE yafray -r ${RENDERREGION[$COUNTER]} $XMLSOURCE 1>>$PATHANDFILENAME$COUNTER.log.txt 2>&1 &"
      ssh $NODE yafray -r ${RENDERREGION[$COUNTER]} $XMLSOURCE 1>>$PATHANDFILENAME-$COUNTER.log 2>&1 & # The process is spawned on a remote node.
   fi

   
   sleep 2      # Gives time for the read of the XML source
   COUNTER=`expr $COUNTER + 1`
done
wait


echo "Compiling image..."
# Now that all our tiles have been rendered, let's piece them all together using ImageMagick
# 1. Crop tiles
if [ `expr $COLUMNS % 2` = 0 ]  # If we have an even number of nodes...
then   # Even number of nodes
   COUNTER=0
   while [ $COUNTER -lt $COLUMNS ]       
   do
      CROPXOFFSET=`echo $COUNTER $COLUMNPIXELRANGE | awk '{print ($1 * $2)}'`
      echo "CROPXOFFSET=$CROPXOFFSET"
      #echo "convert ${OUTPUTCOLUMNPATHS[$COUNTER]} -crop $COLUMNPIXELRANGEx$RESY+$CROPXOFFSET+0 ${OUTPUTCOLUMNPATHS[$COUNTER]}"
      convert ${OUTPUTCOLUMNPATHS[$COUNTER]} -crop "$COLUMNPIXELRANGE"x$RESY+$CROPXOFFSET+0 ${OUTPUTCOLUMNPATHS[$COUNTER]}
      COUNTER=`expr $COUNTER + 1`
   done
else   # We have an odd number of nodes
   #echo "Odd number of nodes, special processing required to set crop images."
   COUNTER=0   
   while [ $COUNTER -lt `expr $COLUMNS - 1` ]
   do
      if [ $COUNTER = `expr $COLUMNS - 2` ]   
      then
         CROPXOFFSET=`echo $COUNTER $ODDCOLUMNPIXELRANGE | awk '{print ($1 * $2)}'`
         echo "CROPXOFFSET=$CROPXOFFSET"
         #echo "convert ${OUTPUTCOLUMNPATHS[$COUNTER]} -crop $COLUMNPIXELRANGEx$RESY+$CROPXOFFSET+0 ${OUTPUTCOLUMNPATHS[$COUNTER]}"
         convert ${OUTPUTCOLUMNPATHS[$COUNTER]} -crop `expr $ODDCOLUMNPIXELRANGE / 2`x$RESY+$CROPXOFFSET+0 ${OUTPUTCOLUMNPATHS[$COUNTER]}
         
         CROPXOFFSET=`echo $COUNTER $ODDCOLUMNPIXELRANGE | awk '{print (($1 * $2) + ($2 / 2))}'`
         echo "CROPXOFFSET=$CROPXOFFSET"
         #echo "convert ${OUTPUTCOLUMNPATHS[$COUNTER]} -crop $COLUMNPIXELRANGEx$RESY+$CROPXOFFSET+0 ${OUTPUTCOLUMNPATHS[$COUNTER]}"
         
         COUNTER=`expr $COUNTER + 1`
         convert ${OUTPUTCOLUMNPATHS[$COUNTER]} -crop "$ODDCOLUMNPIXELRANGE"x$RESY+$CROPXOFFSET+0 ${OUTPUTCOLUMNPATHS[$COUNTER]}
         
      else
         CROPXOFFSET=`echo $COUNTER $ODDCOLUMNPIXELRANGE | awk '{print ($1 * $2)}'`
         echo "CROPXOFFSET=$CROPXOFFSET"
         #echo "convert ${OUTPUTCOLUMNPATHS[$COUNTER]} -crop $COLUMNPIXELRANGEx$RESY+$CROPXOFFSET+0 ${OUTPUTCOLUMNPATHS[$COUNTER]}"
         convert ${OUTPUTCOLUMNPATHS[$COUNTER]} -crop "$ODDCOLUMNPIXELRANGE"x$RESY+$CROPXOFFSET+0 ${OUTPUTCOLUMNPATHS[$COUNTER]}         
      fi
      
      COUNTER=`expr $COUNTER + 1`
   done
   
fi
# 2. Now add all the tiles together for the final image
# Make new file based on original output path
convert -size "$RESX"x"$RESY" xc:skyblue $OUTPUTFILE
# Add all the tiles to the main image
if [ `expr $COLUMNS % 2` = 0 ]  # If we have an even number of nodes...
then   # Even number of nodes
   COUNTER=0
   while [ $COUNTER -lt $COLUMNS ]       
   do
      XOFFSET=`echo $COUNTER $COLUMNPIXELRANGE | awk '{print ($1 * $2)}'`
      #echo "CROPXOFFSET=$XOFFSET"
      #echo "composite -geometry +"$XOFFSET"+0 ${OUTPUTCOLUMNPATHS[$COUNTER]} $OUTPUTFILE $OUTPUTFILE"
      composite -geometry +"$XOFFSET"+0 ${OUTPUTCOLUMNPATHS[$COUNTER]} $OUTPUTFILE $OUTPUTFILE
      COUNTER=`expr $COUNTER + 1`
   done
else   # Odd number of nodes
   #echo "Odd number of nodes, special processing required to make composite image."
   COUNTER=0   
   while [ $COUNTER -lt `expr $COLUMNS - 1` ]
   do
      if [ $COUNTER = `expr $COLUMNS - 2` ]   
      then
         XOFFSET=`echo $COUNTER $ODDCOLUMNPIXELRANGE | awk '{print ($1 * $2)}'`
         echo "XOFFSET=$XOFFSET"
         #echo "composite -geometry +"$XOFFSET"+0 ${OUTPUTCOLUMNPATHS[$COUNTER]} $OUTPUTFILE $OUTPUTFILE"
         composite -geometry +"$XOFFSET"+0 ${OUTPUTCOLUMNPATHS[$COUNTER]} $OUTPUTFILE $OUTPUTFILE
         
         XOFFSET=`echo $COUNTER $ODDCOLUMNPIXELRANGE | awk '{print (($1 * $2) + ($2 / 2))}'`
         COUNTER=`expr $COUNTER + 1`
         echo "XOFFSET=$XOFFSET"
         #echo "composite -geometry +"$XOFFSET"+0 ${OUTPUTCOLUMNPATHS[$COUNTER]} $OUTPUTFILE $OUTPUTFILE"
         composite -geometry +"$XOFFSET"+0 ${OUTPUTCOLUMNPATHS[$COUNTER]} $OUTPUTFILE $OUTPUTFILE
               
      else
         XOFFSET=`echo $COUNTER $ODDCOLUMNPIXELRANGE | awk '{print ($1 * $2)}'`
         echo "XOFFSET=$XOFFSET"
         #echo "composite -geometry +"$XOFFSET"+0 ${OUTPUTCOLUMNPATHS[$COUNTER]} $OUTPUTFILE $OUTPUTFILE"
         composite -geometry +"$XOFFSET"+0 ${OUTPUTCOLUMNPATHS[$COUNTER]} $OUTPUTFILE $OUTPUTFILE
   
      fi
      
      COUNTER=`expr $COUNTER + 1`      
   done
fi


# Change the output path in the XML back to its original
NEWXML=`/usr/bin/xmlstarlet ed -P -u "/scene/render/outfile/@value" -v $OUTPUTFILE $XMLSOURCE`
echo "$NEWXML" > $XMLSOURCE
# Remove the "<?xml version="1.0"?>" that xmlstarlet automatically adds.
OLDXML=`cat $XMLSOURCE`
NEWXML=`echo "$OLDXML" | sed '1d'`
echo "$NEWXML" > $XMLSOURCE
   
   

ENDTIME=`date +%s`
TOTALTIME=`expr \$ENDTIME - \$STARTTIME`
echo "Total rendering time: $TOTALTIME seconds."
exit
mattbatten
 
Posts: 1
Joined: Tue Jan 11, 2005 10:19 am

Return to News & Discussion



Who is online

Users browsing this forum: No registered users and 2 guests

cron