[GitHub] zeppelin pull request #2474: [ZEPPELIN-2753] Basic Implementation of IPython...

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin pull request #2474: [ZEPPELIN-2753] Basic Implementation of IPython...

asfgit
GitHub user zjffdu opened a pull request:

    https://github.com/apache/zeppelin/pull/2474

    [ZEPPELIN-2753] Basic Implementation of IPython Interpreter

    ### What is this PR for?
    This is the first step for implement IPython Interpreter in Zeppelin.  I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well.  
   
    ### What type of PR is it?
    [Feature ]
   
    ### Todos
    * [ ] - Task
   
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-2753
   
    ### How should this be tested?
    Unit is added.
   
    ### Screenshots (if appropriate)
    Verify bokeh in IPython Interpreter
    ![image](https://user-images.githubusercontent.com/164491/27999716-756d749e-6552-11e7-90bb-4c6b08f4ab5c.png)
   
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2753

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/2474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2474
   
----
commit e2fbf75e0bf3a814122fa51ea3a88da2823f0487
Author: Jeff Zhang <[hidden email]>
Date:   2017-07-02T07:51:14Z

    [ZEPPELIN-2753] Basic Implementation of IPython Interpreter

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    @Leemoonsoo @1ambda @felixcheung @prabhjyotsingh Please help review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Thanks for testing @Leemoonsoo . These special characters are for font color, create ZEPPELIN-2756 for that which might need help from @1ambda


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    @Leemoonsoo Regarding the ZeppelinContext, could you try that again ? It is covered in unit test, so should be able to work.  I force committed it yesterday, the latest commit Id is 956befa4151128b9ee76df2e3eada512ab9e1025


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user Leemoonsoo commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Tested 88a1f2f and ZeppelinContext works well.
   
    Does auto completion supposed to work?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Yes, that works. It is covered in unit test (https://github.com/apache/zeppelin/pull/2474/files#diff-d9c8252d37a199d13b7157e88041c0e9R124).  Also I manually verified it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user Leemoonsoo commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Is it expected completion list?
    ![image](https://user-images.githubusercontent.com/1540981/28252185-32d695a8-6ac9-11e7-9f29-2257700990d0.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Thanks @Leemoonsoo for catching that. This is due to I miss cursor for code completion. Just pushed another commit to fix that and add more unit test for it.
   
    ```
        // there's no completion for 'a.' because it is not recognized by compiler for now.
        String st = "a='hello'\na.";
        completions = interpreter.completion(st, st.length(), getInterpreterContext());
        assertEquals(0, completions.size());
   
        // define a first
        st = "a='hello'";
        result = interpreter.interpret(st, getInterpreterContext());
        assertEquals(InterpreterResult.Code.SUCCESS, result.code());
   
        // now we can get the completion for `a.`
        st = "a.";
        completions = interpreter.completion(st, st.length(), getInterpreterContext());
        // it is differnet for python2 and python3 and may even different for different minor version
        // so only verify it is larger than 20
        assertTrue(completions.size() > 20);
   
        st = "a.ca";
        completions = interpreter.completion(st, st.length(), getInterpreterContext());
        // a.capitalize
        // a.casefold
        assertEquals(2, completions.size());
   
        // cursor is in the middle of code
        st = "a.ca\b='hello";
        completions = interpreter.completion(st, 4, getInterpreterContext());
        // a.capitalize
        // a.casefold
        assertEquals(2, completions.size());
   
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user 1ambda commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Let create a PR to fix the ASCII issue, we can test this PR based on the ASCII PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    @1ambda I did some investtigation on that, and puts some links in https://issues.apache.org/jira/browse/ZEPPELIN-2756, hope it would be helpful for you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Just push another commit to use IPythonInterpreter first, if it is not available, I will fallback it to the old PythonInterpreter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Push another commit to use IPython in PySparkInterpreter if IPython is available , otherwise fallback to the old implementation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Push another commit to support streaming output


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    @Leemoonsoo @1ambda @felixcheung @prabhjyotsingh I made several follow up commit. All the features in this stage is done. Please help review that. I believe it would be a big improvement for python users.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin pull request #2474: [ZEPPELIN-2753] Basic Implementation of IPython...

asfgit
In reply to this post by asfgit
Github user zjffdu closed the pull request at:

    https://github.com/apache/zeppelin/pull/2474


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin pull request #2474: [ZEPPELIN-2753] Basic Implementation of IPython...

asfgit
In reply to this post by asfgit
GitHub user zjffdu reopened a pull request:

    https://github.com/apache/zeppelin/pull/2474

    [ZEPPELIN-2753] Basic Implementation of IPython Interpreter

    ### What is this PR for?
    This is the first step for implement IPython Interpreter in Zeppelin.  I just use the jupyter_client to create and manage the ipython kernel. We don't need to care about python compilation and execution, all the things are delegated to ipython kernel. Ideally all the features of ipython should be available in Zeppelin as well.  
   
    For now, user can use %python.ipython for IPython Interpreter. And if ipython is available, the default python interpreter will use ipython. But user can still set `zeppelin.python.useIPython` as false to enforce to use the old implementation of python interpreter.
   
    Main features:
    * IPython interpreter support
    ** All the ipython features are available, including visualization, ipython magics.
    * ZeppelinContext support
    * Streaming output support
    * Support Ipython in PySpark
   
    Regarding the visualization, ideally all the visualization libraries work in jupyter should also work here.
    In unit test, I only verify the following 3 popular visualization library. could add more later.
    * matplotlib
    * bokeh
    * ggplot
   
    ### What type of PR is it?
    [Feature ]
   
    ### Todos
    * [ ] - Task
   
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-2753
   
    ### How should this be tested?
    Unit test is added.
   
    ### Screenshots (if appropriate)
    Verify bokeh in IPython Interpreter
    ![image](https://user-images.githubusercontent.com/164491/27999716-756d749e-6552-11e7-90bb-4c6b08f4ab5c.png)
   
    Verify matplotlib
    ![image](https://user-images.githubusercontent.com/164491/28046960-e881b28e-6619-11e7-9e1f-7f4662f842f3.png)
   
    Verify ZeppelinContext
   
    ![image](https://user-images.githubusercontent.com/164491/28119378-4212620c-6747-11e7-89d5-3b5e609593ce.png)
   
    Verify Streaming
    ![streaming](https://user-images.githubusercontent.com/164491/28950974-8f92fe1e-78fa-11e7-841f-3174da198bb7.gif)
   
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2753

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/2474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2474
   
----
commit 78fde23c39a9fef36cea3d074b09a4748bdc17c9
Author: Jeff Zhang <[hidden email]>
Date:   2017-07-02T07:51:14Z

    [ZEPPELIN-2753] Basic Implementation of IPython Interpreter

commit bb4d35451e3a7e83d20a2a7f66f303f2258bdd16
Author: Jeff Zhang <[hidden email]>
Date:   2017-07-17T00:29:59Z

    Add cursor for completion

commit 3df231497a2b4b4bd0e3703151d25a0b0a932073
Author: Jeff Zhang <[hidden email]>
Date:   2017-07-27T11:10:27Z

    use IPython first, otherwise fallback to the old PythonInterpreter

commit 6d1f69bb74639ebe7c826a2ed807acfa8b02619e
Author: Jeff Zhang <[hidden email]>
Date:   2017-07-28T07:49:29Z

    IPython Support for PySparkInterpreter

commit 3179e9abb71ea63051c1dbb918276c9c24aba6e7
Author: Jeff Zhang <[hidden email]>
Date:   2017-07-30T09:54:23Z

    Support streaming output

commit 4c129c66e0eca39157a742d20bed7773ffedc0d5
Author: Jeff Zhang <[hidden email]>
Date:   2017-08-04T02:01:34Z

    add more docs

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user Leemoonsoo commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    Tried last commit.
   
    I installed jupyter, IPython, ipykernel, grpcio in my conda environment and i configured my python interpreter like
    ![image](https://user-images.githubusercontent.com/1540981/29084421-c7df43a8-7c20-11e7-9299-2d7ab88d17fd.png)
   
    And then i'm getting `No module named py4j.java_gateway` error on first `%python.ipython` execution
   
    ![image](https://user-images.githubusercontent.com/1540981/29084458-da47f3b4-7c20-11e7-8ab7-6c281f0c3d54.png)
   
    And second execution, `z` is not available.
   
    ![image](https://user-images.githubusercontent.com/1540981/29084488-f49a8358-7c20-11e7-8b99-47605981faa3.png)
   
    Looks like py4j library is not in python path. @zjffdu Do you have any idea on this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] zeppelin issue #2474: [ZEPPELIN-2753] Basic Implementation of IPython Interp...

asfgit
In reply to this post by asfgit
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/2474
 
    @Leemoonsoo I have fixed the issue, could you try it again ? Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Loading...