Quantcast

[PATCH] Proposal for a new external template: gnuplot

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH] Proposal for a new external template: gnuplot

Koji Yokota
Hi all,

The attached is a proposal for a new external template for gnuplot, a
graph plotting program. I have been using it for my personal use for a
while, but I thought it may be better included in the main body as it is
convenient.

With this template, users can directly edit the gnuplot source code from
lyx, whereas its consequent graphics is automatically included in outputs.

Although the attached script can generate other graphics formats as
well, in the purpose of external template, it is only used for
generation of an EPS graphics file from a gnuplot source code. If
necessary, users can add additional converters for each graphics format.

I think it should be better checked by professionals, but I hope it is
useful.

How do you think?

Best regards,

Koji


--- lib/configure.py.orig 2008-05-06 22:20:27.000000000 +0900
+++ lib/configure.py 2008-05-15 01:37:48.000000000 +0900
@@ -244,6 +244,10 @@
     checkViewer('a FEN viewer and editor', ['xboard -lpf $$i -mode EditPosition'],
         rc_entry = [r'\Format fen        fen     FEN                    "" "%%" "%%" ""'])
     #
+    checkViewer('a Gnuplot viewer and editor', ['emacs', 'emcws', 'xemacs', 'gvim', 'kedit', 'kwrite', \
+        'kate', 'nedit', 'gedit', 'notepad'],
+        rc_entry = [r'\Format gnuplot    gnuplot Gnuplot                "" "%%" "%%" "vector"'])
+    #
     path, iv = checkViewer('a raster image viewer', ['xv', 'kview', 'gimp-remote', 'gimp'])
     path, ie = checkViewer('a raster image editor', ['gimp-remote', 'gimp'])
     addToRC(r'''\Format bmp        bmp     BMP                    "" "%s" "%s" ""
@@ -499,6 +503,10 @@
     checkProg('a Noteedit -> LilyPond converter', ['noteedit --export-lilypond $$i'],
         rc_entry = [ r'\converter noteedit   lilypond   "%%" ""', ''])
     #
+    checkProg('a Gnuplot -> EPS converter', ['gnuplot'],
+        rc_entry = [ r'\converter gnuplot    eps        "python -tt $$s/scripts/gnuplot.py $$i $$o" ""', ''])
+    #
+    #
     # FIXME: no rc_entry? comment it out
     # checkProg('Image converter', ['convert $$i $$o'])
     #
--- lib/external_templates.orig 2008-05-15 01:43:46.000000000 +0900
+++ lib/external_templates 2008-05-15 01:44:46.000000000 +0900
@@ -314,3 +314,58 @@
  UpdateResult "$$Tempname"
  FormatEnd
 TemplateEnd
+
+Template gnuplot
+        GuiName "gnuplot: $$AbsOrRelPathParent$$Basename"
+        HelpText
+                Create plots and graphics with Gnuplot
+
+ 'gnuplot' binary must be found in path for it
+ to work correctly.
+ If you use emacs as an editor, you may want
+ to use it in the gnuplot mode.
+        HelpTextEnd
+        InputFormat gnuplot
+        FileFilter "*.{gnuplot,plt,plot,gnu}"
+        AutomaticProduction true
+        Transform Rotate
+        Transform Resize
+        Transform Clip
+        Transform Extra
+        Format LaTeX
+                Requirement "graphicx"
+                TransformOption Rotate RotationLatexOption
+                TransformOption Resize ResizeLatexOption
+                TransformOption Clip   ClipLatexOption
+                TransformOption Extra  ExtraOption
+                Option Arg "[$$Extra,$$Rotate,$$Resize,$$Clip]"
+                Product "\\includegraphics$$Arg{$$AbsOrRelPathMaster$$Basename.e
+ps}"
+                UpdateFormat eps
+                UpdateResult "$$AbsPath$$Basename.eps"
+                ReferencedFile latex "$$AbsOrRelPathMaster$$Basename.eps"
+                ReferencedFile dvi   "$$AbsPath$$Basename.eps"
+        FormatEnd
+        Format PDFLaTeX
+                Requirement "graphicx"
+                TransformOption Rotate RotationLatexOption
+                TransformOption Resize ResizeLatexOption
+                TransformOption Clip   ClipLatexOption
+                TransformOption Extra  ExtraOption
+                Option Arg "[$$Extra,$$Rotate,$$Resize,$$Clip]"
+                Product "\\includegraphics$$Arg{$$AbsOrRelPathMaster$$Basename.e
+ps}"
+                UpdateFormat eps
+                UpdateResult "$$AbsPath$$Basename.eps"
+                ReferencedFile pdflatex "$$AbsOrRelPathMaster$$Basename.eps"
+        FormatEnd
+        Format Ascii
+                Product "[gnuplot: $$FName]"
+        FormatEnd
+        Format DocBook
+                Product "[gnuplot: $$FName]"
+        FormatEnd
+        Format LinuxDoc
+                Product "[gnuplot: $$FName]"
+        FormatEnd
+TemplateEnd
--- lib/scripts/gnuplot.py.orig 2008-05-15 01:46:32.000000000 +0900
+++ lib/scripts/gnuplot.py 2008-05-11 03:11:09.000000000 +0900
@@ -0,0 +1,123 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# file gnuplot.py
+# This file is part of LyX, the document processor.
+# Licence details can be found in the file COPYING.
+
+# \author Koji Yokota
+
+# Full author contact details are available in file CREDITS.
+
+# This script creates a graphic file from a GnuPlot source file.
+# It is aimed to be called from external templates.
+# For it to work correctly, gnuplot must be found in path.
+#
+# Usage: python gnuplot.py [-t type [-o option]] \
+#                          <GnuPlot source file> <EPS output file>
+#            type:   terminal type
+#            option: comma-separated options for a given terminal type
+#        Default values are:
+#            type:   "postscript"
+#            option: "eps enhanced color"
+#                    (It will be reset to null once type is specified)
+
+import re, sys, os, getopt, tempfile
+
+class Error: pass
+
+# Get options and arguments
+opts, args = getopt.getopt(sys.argv[1:], "t:o:")
+try:
+    if (len(args) != 2):
+        raise Error()
+except Error:
+    print '\n' + sys.argv[0]
+    print '\nThis script outputs a graphics file from a Gnuplot source file.\n'
+    print 'Usage:\n    python ' + sys.argv[0] + ' [-t type [-o option]] \\'
+    print '                      <gnuplot source file> <output graphics file>\n'
+    print '        type:   terminal type (see "help terminal" in gnuplot for'
+    print '                available values)'
+    print '        option: comma-separated options for a given terminal type'
+    print '                (see "help terminal <terminal type>" in gnuplot for'
+    print '                 available values)'
+    print '\n        Default values are:'
+    print '            type:   "postscript"'
+    print '            option: "eps enhanced color" (It will be reset to null'
+    print '                     once type is specified)\n'
+    sys.exit()
+
+# Set output graphic type and options
+# It's very likely that terminal type and its options are set for review
+# purpose (e.g. x11) or else. This script automates specification of
+# output format.
+terminal_type = 'postscript'
+terminal_option = ['eps', 'enhanced', 'color']
+type_specified = 0
+option_specified = 0
+for o,v in opts:
+ if o == '-t':
+ terminal_type = v
+ type_specified = 1
+ if o == '-o':
+ terminal_option = v.split(',')
+ option_specified = 1
+if type_specified and not option_specified:
+ terminal_option = []
+
+# Set "set terminal" and "set output" lines in the gnuplot source file
+terminal_line = 'set terminal ' + terminal_type
+if terminal_option != []:
+ for o in terminal_option:
+ terminal_line = terminal_line + ' ' + o
+output_line = 'set output "' + args[1] + '"'
+
+# Definition of strings to be replaced
+find_terminal = re.compile(r"(^\s*set\s+terminal.*$)",re.IGNORECASE)
+find_output = re.compile(r"(^\s*set\s+output.*$)",re.IGNORECASE)
+find_comment = re.compile(r"(^#|^\s*$)",re.IGNORECASE)
+
+# comment_out function comments out 'set terminal...' and 'set output...'
+# lines and outputs other lines as is
+def comment_out(line):
+    if find_terminal.search(line):
+        print find_terminal.sub(r'# \1', line[:-1])
+        count_terminal = 1
+    elif find_output.search(line):
+        print find_output.sub(r'# \1', line[:-1])
+        count_output = 1
+    else:
+        print line[:-1]
+
+# Flag to show whether head comment part is finished
+comment_ended = 0
+
+# Open a temporary file for a modified source
+tmp_filename = tempfile.mktemp()
+tmp_file = open(tmp_filename, 'w')
+sys.stdout = tmp_file
+
+# Replace and output lines one by one
+for line in open(args[0]):
+    if comment_ended == 1:
+        comment_out(line)
+    else:
+        if not find_comment.search(line):
+            # This is the first line after the comment part
+            # Insert new 'set terminal' and 'set output'
+    print terminal_line
+    print output_line
+            comment_out(line)
+            comment_ended = 1
+        else:
+            # Comment part still continues, just output lines
+            print line[:-1]
+
+tmp_file.close()
+
+# Execute gnuplot to generate a graphics file from the source
+try:
+    os.system('gnuplot %(tmp_filename)s' % vars())
+finally:
+    # Remove the temporary file
+    os.remove(tmp_filename)
\ No newline at end of file
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

José Matos
On Thursday 15 May 2008 16:14:29 Koji Yokota wrote:
> I think it should be better checked by professionals, but I hope it is
> useful.
>
> How do you think?

Historically the problem with gnuplot was the fact that is has not any
sandboxing scheme.

So it should be possible to send you a gnuplot file that erases your home area
if you run gnuplot over it.

Example (this is specific to unix/linux but it can be adapted to other
systems):

system("rm -rf ~")

AFAIK the situation has not changed in the 4.s series.

Thanks for your suggestion. :-)

> Best regards,
>
> Koji

--
José Abílio
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Christian Ridderström
On Thu, 15 May 2008, José Matos wrote:

> On Thursday 15 May 2008 16:14:29 Koji Yokota wrote:
>> I think it should be better checked by professionals, but I hope it is
>> useful.
>>
>> How do you think?

> Example (this is specific to unix/linux but it can be adapted to other
> systems):
>
> system("rm -rf ~")
>
> AFAIK the situation has not changed in the 4.s series.

How about the following suggestion. Koji can upload the template to the
wiki and create a wiki page that describes how to use it. The page should
also warn of the dangerous case described above.

/Christian

--
Christian Ridderström, +46-8-768 39 44               http://www.md.kth.se/~chr
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Koji Yokota
José Matos wrote:
 > Historically the problem with gnuplot was the fact that is has not any
 > sandboxing scheme.
 >
 > So it should be possible to send you a gnuplot file that erases your
home area
 > if you run gnuplot over it.
 >
 > Example (this is specific to unix/linux but it can be adapted to other
 > systems):
 >
 > system("rm -rf ~")

I see. It really should be used at users' own risk.

[hidden email] wrote:
> How about the following suggestion. Koji can upload the template to the
> wiki and create a wiki page that describes how to use it. The page
> should also warn of the dangerous case described above.

That's a good idea. If there are no other problems, I can do so.

Koji
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Christian Ridderström
On Fri, 16 May 2008, Koji Yokota wrote:

> José Matos wrote:
>>  Historically the problem with gnuplot was the fact that is has not any
>>  sandboxing scheme.
>>
>>  So it should be possible to send you a gnuplot file that erases your
> home area
>>  if you run gnuplot over it.
>>
>>  Example (this is specific to unix/linux but it can be adapted to other
>>  systems):
>>
>>  system("rm -rf ~")
>
> I see. It really should be used at users' own risk.
>
> [hidden email] wrote:
>>  How about the following suggestion. Koji can upload the template to the
>>  wiki and create a wiki page that describes how to use it. The page should
>>  also warn of the dangerous case described above.
>
> That's a good idea. If there are no other problems, I can do so.
You could look at this page for inspiration:
  http://wiki.lyx.org/Examples/IncludeExternalProgramListing

Caveat: It's a random page I picked that happened to include 'external'
and 'template'...

The most difficult thing here is coming up with a good name for the wiki
page. How about:

  Examples/UsingGnuplotWithLyX

i.e.

  http://wiki.lyx.org/Examples/UsingGnuplotWithLyX

(we can rename it later of course!)

/Christian

PS Let me know if you need help with uploading files. I'll send you the
upload password separately.

--
Christian Ridderström, +46-8-768 39 44               http://www.md.kth.se/~chr
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Helge Hafting-3
In reply to this post by José Matos
José Matos wrote:

> On Thursday 15 May 2008 16:14:29 Koji Yokota wrote:
>  
>> I think it should be better checked by professionals, but I hope it is
>> useful.
>>
>> How do you think?
>>    
>
> Historically the problem with gnuplot was the fact that is has not any
> sandboxing scheme.
>
> So it should be possible to send you a gnuplot file that erases your home area
> if you run gnuplot over it.
>
> Example (this is specific to unix/linux but it can be adapted to other
> systems):
>
> system("rm -rf ~")
>  
Ouch.
Could this be solved by having LyX refusing to use gnuplot file
containing the "system" command? Or is it much more complicated?

Helge Hafting
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Koji Yokota
Helge Hafting wrote:
> Ouch.
> Could this be solved by having LyX refusing to use gnuplot file
> containing the "system" command? Or is it much more complicated?
>
> Helge Hafting
>

Yeah right... If only shell execution can be a security hall (is this
right?), we can comment out the related lines from the source file by
the python script. Anyway the script scans over the source file, doing
so is rather straightforward.

So, how about modifying the gnuplot.py in a way that, if it finds out
the expression "^\s*system" or "^\s*\!", it comments out the related
lines. It may break the consistency of the source, therefore user may
not obtain the correct output, it may be more flexible than stopping the
execution of gnuplot itself. We simply warn users in the help
description, that the gnuplot source may not work correctly if "system"
or "!" lines are included as they will be skipped.

Koji
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Koji Yokota
> if it finds out
> the expression "^\s*system" or "^\s*\!", it comments out the related
> lines.

In addition, "load" function must be skipped because its content is not
known to gnuplot.py. Also, we must consider the case that multiple
commands exist in one line.

Koji
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Alfredo Braunstein-2
Koji Yokota wrote:

>> if it finds out
>> the expression "^\s*system" or "^\s*\!", it comments out the related
>> lines.
>
> In addition, "load" function must be skipped because its content is not
> known to gnuplot.py. Also, we must consider the case that multiple
> commands exist in one line

It's more complicated than that, e.g.

gnuplot> plot "<rm -rf /"

Maybe some solution involving a chroot?

A/


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Jean-Marc Lasgouttes
In reply to this post by Koji Yokota
Koji Yokota <[hidden email]> writes:
> Yeah right... If only shell execution can be a security hall (is this
> right?), we can comment out the related lines from the source file by
> the python script. Anyway the script scans over the source file, doing
> so is rather straightforward.

The only good solution would be an execution mode of gnuplot that
disables calls to system. Last time I checked, it did not exist.

JMarc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Koji Yokota
Alfredo Braunstein wrote:
 > It's more complicated than that, e.g.
 >
 > gnuplot> plot "<rm -rf /"

I could never imagine gnuplot accepts even such expression :o

 > Maybe some solution involving a chroot?

Jean-Marc Lasgouttes wrote:
 > The only good solution would be an execution mode of gnuplot that
 > disables calls to system. Last time I checked, it did not exist.

For completeness, it seems there is no other workaround. It is really
sad that we have to give up this feature, as there will be many people
who are using gnuplot together with lyx or latex. I saw some programs
like Excel pops up an alert when user tries to open a spreadsheet
containing macros. Is it too risky to do the same thing? Say,

        - Whenever gnuplot.py is called it pops up an alert, saying
           it is exposing the system to some risk (and explain it well)
           and urge user to check the gnuplot code
        - gnuplot temporary file is made in a newly created directory
           and gnuplot is executed after chroot'ing there
        - gnuplot.py stops executing if user is using it with root
           privileges

I haven't checked the feasibility of the code, but once this is done, at
least files on the disk will not be modified. But of course, depending
on the privileges the user has, if malicious code is *carelessly*
executed, it may be able to do some weired things like shutting down the
computer or else.

I'm not sure if this level of risk is acceptable or not...

Koji
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Jean-Marc.Lasgouttes
Koji Yokota <[hidden email]> writes:

> - Whenever gnuplot.py is called it pops up an alert, saying
>           it is exposing the system to some risk (and explain it well)
>           and urge user to check the gnuplot code

People tend to click OK without reading.

> - gnuplot temporary file is made in a newly created directory
>           and gnuplot is executed after chroot'ing there

But what if gnuplot needs to access some files?

JMarc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Koji Yokota
Jean-Marc Lasgouttes wrote:
> People tend to click OK without reading.

Yes. So, the severity of damage after careless click should not be too
large, at least. The risk in that case would be limited to what users
can do with shell's built-in commands. I wonder if this is still too risky.

>
>> - gnuplot temporary file is made in a newly created directory
>>           and gnuplot is executed after chroot'ing there
>
> But what if gnuplot needs to access some files?

For drawing purpose, gnuplot itself needs only a few files (a binary and
a driver in *BSD), which probably is not too costly to copy under
chroot. If it uses a data file, the filename can be obtained from the
source. But we may also have to impose some limitations, e.g. load of
another gnuplot source file from the source, which we must be content with.

Koji

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Pavel Sanda
In reply to this post by Koji Yokota
Koji Yokota wrote:

> Alfredo Braunstein wrote:
> > It's more complicated than that, e.g.
> >
> > gnuplot> plot "<rm -rf /"
>
> I could never imagine gnuplot accepts even such expression :o
>
> > Maybe some solution involving a chroot?
>
> Jean-Marc Lasgouttes wrote:
> > The only good solution would be an execution mode of gnuplot that
> > disables calls to system. Last time I checked, it did not exist.
>
> For completeness, it seems there is no other workaround. It is really sad
> that we have to give up this feature, as there will be many people who are
> using gnuplot together with lyx or latex.

i think that the correct solution would be to request gnuplot devs to add
some special parameter from commandline which disallows to execute any
external programs, either from system() or the plot "<rm -rf /" issues.

pavel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

José Matos
On Monday 19 May 2008 18:23:55 Pavel Sanda wrote:
>
> i think that the correct solution would be to request gnuplot devs to add
> some special parameter from commandline which disallows to execute any
> external programs, either from system() or the plot "<rm -rf /" issues.

  IIRC last time we visited this subject Angus asked this on gnuplot devel
list and they were unsympathetic about this proposal.

> pavel

--
José Abílio
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Pavel Sanda
José Matos wrote:
> > i think that the correct solution would be to request gnuplot devs to add
> > some special parameter from commandline which disallows to execute any
> > external programs, either from system() or the plot "<rm -rf /" issues.
>
>   IIRC last time we visited this subject Angus asked this on gnuplot devel
> list and they were unsympathetic about this proposal.

aha. but there is nothing to lose if Koji ask once again ;)
pavel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Jean-Marc Lasgouttes
Pavel Sanda <[hidden email]> writes:

> José Matos wrote:
>> > i think that the correct solution would be to request gnuplot devs to add
>> > some special parameter from commandline which disallows to execute any
>> > external programs, either from system() or the plot "<rm -rf /" issues.
>>
>>   IIRC last time we visited this subject Angus asked this on gnuplot devel
>> list and they were unsympathetic about this proposal.
>
> aha. but there is nothing to lose if Koji ask once again ;)

But then we'll need a way to check that the version of gnuplot is
recent enough.

JMarc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Pavel Sanda
Jean-Marc Lasgouttes wrote:
> > aha. but there is nothing to lose if Koji ask once again ;)
>
> But then we'll need a way to check that the version of gnuplot is
> recent enough.

this is easy, old version of gnuplot fail to run if any else parameter
than file is given. hackish, yes ;)

pavel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Helge Hafting-3
In reply to this post by Jean-Marc.Lasgouttes
Jean-Marc Lasgouttes wrote:
> Koji Yokota <[hidden email]> writes:
>
>> - Whenever gnuplot.py is called it pops up an alert, saying
>>           it is exposing the system to some risk (and explain it well)
>>           and urge user to check the gnuplot code
>
> People tend to click OK without reading.

If you have sympathy for the mindless, have LyX refuse the
possibly dangerous gnuplot file instead of merely warning.
I.e. let the popup say "this gnuplot file uses the dangerous
'system' command so it was skipped." The user who clicks that
one away get a document with a blank figure.


Helge Hafting
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Proposal for a new external template: gnuplot

Jean-Marc Lasgouttes
Helge Hafting <[hidden email]> writes:

> If you have sympathy for the mindless, have LyX refuse the
> possibly dangerous gnuplot file instead of merely warning.
> I.e. let the popup say "this gnuplot file uses the dangerous
> 'system' command so it was skipped." The user who clicks that
> one away get a document with a blank figure.

I would be surprised if we could really spot dangerous scripts easily.

JMarc
12
Loading...