Remote UI BoF V. Stirbu Internet-Draft Nokia Expires: December 31, 2005 June 29, 2005 LRDP: The Lightweight Remote Display Protocol draft-stirbu-lrdp-00 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on December 31, 2005. Copyright Notice Copyright (C) The Internet Society (2005). Abstract This memo describes an application layer protocol for a framework that enables accessing graphical user interface (GUI) applications remotely, so that devices with different form factors and UI capabilities can scale and adapt the exported UI to their local platform UI look and feel (LAF). Specified in the Extensible Markup Language (XML), the protocol defines generic user interface (UI) remoting operations. Requirements Language Stirbu Expires December 31, 2005 [Page 1] Internet-Draft LRDP June 2005 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [1]. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 General Concepts . . . . . . . . . . . . . . . . . . . . . 3 1.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 The UI Remoting Framework . . . . . . . . . . . . . . . . 4 1.3.1 Discovery-Transport Layer . . . . . . . . . . . . . . 4 1.3.2 UI Remoting Elements Layer . . . . . . . . . . . . . . 5 1.3.3 Application Specific Layer . . . . . . . . . . . . . . 5 1.4 Architecture Paradigm . . . . . . . . . . . . . . . . . . 5 1.4.1 UI Model . . . . . . . . . . . . . . . . . . . . . . . 5 1.4.2 Session Setup . . . . . . . . . . . . . . . . . . . . 6 1.4.3 Remoting the User Interface . . . . . . . . . . . . . 6 2. Exchange Description . . . . . . . . . . . . . . . . . . . . . 6 2.1 uiInitialisation Format . . . . . . . . . . . . . . . . . 6 2.2 uiFragment Format . . . . . . . . . . . . . . . . . . . . 7 2.3 uiUpdate Format . . . . . . . . . . . . . . . . . . . . . 7 2.4 uiEvent Format . . . . . . . . . . . . . . . . . . . . . . 7 3. Formal XML Syntax . . . . . . . . . . . . . . . . . . . . . . 7 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 5. Security Considerations . . . . . . . . . . . . . . . . . . . 9 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 9 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 9 7.1 Normative References . . . . . . . . . . . . . . . . . . . 9 7.2 Informative References . . . . . . . . . . . . . . . . . . 10 Author's Address . . . . . . . . . . . . . . . . . . . . . . . 10 A. UI Server Consideration . . . . . . . . . . . . . . . . . . . 10 B. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 B.1 Initialisation . . . . . . . . . . . . . . . . . . . . . . 11 B.2 Client Updates . . . . . . . . . . . . . . . . . . . . . . 12 B.3 Client Event . . . . . . . . . . . . . . . . . . . . . . . 13 B.4 Server Update . . . . . . . . . . . . . . . . . . . . . . 14 Intellectual Property and Copyright Statements . . . . . . . . 15 Stirbu Expires December 31, 2005 [Page 2] Internet-Draft LRDP June 2005 1. Introduction The protocol outlined in this document is based on the functional requirements described in the Internet-Draft Requirements for Scalable Remote Access to Applications . This document presents a framework that enables applications to remote, in a scalable and adaptable way, the user interface (UI) on remote devices using the Lightweight Remote Display Protocol (LRDP). LRDP is a text based protocol based on Extensible Markup Language (XML) 1.0 as described in [2], XML schema notation as described in [3] and [4], and XML namespaces as described in [5]. 1.1 General Concepts A user interface (UI) may be composed of several widgets, wherein a widget is understood as an element of a user interface that displays information or provides a specific way for a user to interact with an application. Widgets may for instance comprise icons, pull-down menus, buttons, selection buttons, progress indicators, on-off checkmarks, scrollbars, windows, window edges, toggle buttons, forms and any other elements for displaying information, inviting, accepting, and responding to user actions. When accessing graphical user interface (GUI) applications remotely, provisions need to be made for customizing the remoted UI to match the characteristic display style (the so-called look & feel) of a device platform. This is of particular importance for ensuring that the user experience is not compromised from one device to another. The IETF architecture can be implemented on multiple platforms with its own UI framework for controlling the rendering of local application(s) UI(s). Each platform has its specific appearance to provide the appropriate user experience. This platform customisation of the UI has to be replicated with the maximum fidelity on the new platform where the UI is remoted. Therefore, the remoting GUI protocol has to ensure a favorable user experience regardless of the underlying viewing platform, so that the user can perceive the GUI as a local application making it intuitively usable. Based on the above criteria, LRDP defines a protocol framework that supports the following features: 1. A means of describing a UI (or parts of a UI) in terms of widgets and styles. 2. A means of communicating (partial) UI updates from a device that hosts an application to a device that is responsible for rendering the UI. 3. A means of describing events triggered through changes in widget state made by the user so that they can be sent to the device that hosts the application. Stirbu Expires December 31, 2005 [Page 3] Internet-Draft LRDP June 2005 Note that LRDP and the XML schema that formally describes LRDP do not specify user interfaces for any particular application or set of applications. This means that it can be applied to many different domain contexts. 1.2 Definitions For clarity, the following definitions are supplied: o UI Servers - The devices that host applications and have the capabilities to export the UI on other devices; they receive UI input from remote users and transmit UI output. o UI Clients - The devices that receive the UI output from a UI server and provide UI input. o UI Remoting Session - The session initiated between UI Servers and UI clients for exchanging UI information to control applications remotely. o UI Remoting Protocol - The types of UI objects, UI events and notifications that are interchanged between a UI client and UI server in order to keep them in sync with each other during a remote UI session. o UI Remoting Framework - The architecture that defines how the protocol should be supported on a UI client and server. 1.3 The UI Remoting Framework The UI Remoting Framework includes the following components grouped on three layers: Discovery-Transport, UI Remoting Elements and Application Specific. +---------------------------+ Application Specific | UI model | +---------------------------+ UI Remoting Elements | LRDP | +---------------+-----------+ Discovery-Transport | session setup | BEEP | +---------------+-----------+ 1.3.1 Discovery-Transport Layer The Discovery-Transport layer includes the components that will implement the remote UI client and server discovery, the remote UI session set up, authentication and connection management, capability negotiation and the transport of UI related information. The Discovery-Transport layer include two components: "session setup" and "transport". o The "session setup" refers to the components that provide discovery features (i.e. for finding remote UI server and clients) and session set up (i.e. negotiating the platform capabilities for Stirbu Expires December 31, 2005 [Page 4] Internet-Draft LRDP June 2005 initiating the UI remoting). o The "transport" refers to the transport protocol used to exchange UI messages between the UI server and the UI client. The "transport" will use the Blocks Extensible Exchange Protocol Core (BEEP) [6]. 1.3.2 UI Remoting Elements Layer The UI Remoting Elements layer define the basic operations needed in order to export the UI and to keep the UI synchronised with the application state. 1.3.3 Application Specific Layer The Application Specific layer defines widgets, widget properties and styling properties that are specific to an application. This layer includes the UI Model component, representing a tree of widgets that make-up the user interface plus additional meta-info about the UI. 1.4 Architecture Paradigm +----------------------+ +----------------------+ | UI Server | | UI Client | | +------------------+ | | +------------------+ | | | Application | | | | Host | | | | +------------+ | | | | Windowing System | | | | |Application | | | | +------------------+ | | | | Logic | | | | +------------------+ | | | +------------+ | | | | Viewer | | | | +------------+ | | | | +------------+ | | | | | UI Model | | | | | | UI Model | | | | | +------------+ | | | | +------------+ | | | +------------------+ | | +------------------+ | | +---------+ +------+ | UI update | +------+ +---------+ | | | Session | | LRDP |<------------->| LRDP | | Session | | | | Setup | | |---------------| | | Setup | | | +---------+ +------+ | UI events | +------+ +---------+ | | | | | +----------------------+ +----------------------+ 1.4.1 UI Model The UI Model represents a tree of widgets that make-up the user interface plus additional meta-info about the UI. At the top level of abstraction, the UI Model is seen as the Viewport in the MVC model [7]; the model (application states) and the Controller (application specific event handlers) are defined within the "Application Logic". Stirbu Expires December 31, 2005 [Page 5] Internet-Draft LRDP June 2005 1.4.2 Session Setup The Session Setup role is to discover, find matches and initiate UI sessions between UI Server and UI Client devices. We define a match between a UI Server and a UI Client when the UI Server supports one or more UI description languages that are supported by the UI Client platform. The Session Setup can use the existing IETF session model, encompassing session descriptions using Session Description Protocol (SDP) [8] or successors and the Session Initiation Protocol (SIP) [9], or UPnP Remote UI framework [10]. 1.4.3 Remoting the User Interface The UI Remoting Framework maintains two copies of the UI Model: the mastercopy in the Application and a replica in the Viewer. The UI Models are synchronised using the LRDP "UI update" and "UI events" channels. The "UI update" channel is used to send the initial UI Model and subsequent updates from the Application to the viewer, and the "UI events" channel is used to sent back to the Application the events triggered by the user interaction with the UI. 2. Exchange Description This section describes the update and event exchanges of the protocol. The descriptions contained within this section refer to XML elements and attributes and their relation to the exchange of data within the protocol. These descriptions also contain specifications outside the scope of the formal XML syntax. Therefore, this section will use the terms defined in RFC 2119 [1] to describe the specifications outside the scope of the formal XML syntax. While reading this section, please reference Section 3 for details on the formal XML syntax. 2.1 uiInitialisation Format The element contains the full description of the user interface in terms of widgets and styles. The first message sent by an application having an UI that doesn't change its structure at run-time MUST be the message. During an UI Remoting Session only one message is sent. All immediate child elements of the element MUST be namespace-qualified and MUST belong to the namespaces that were negotiated during the Session Setup step. Because UI description documents can be quite large, it is Stirbu Expires December 31, 2005 [Page 6] Internet-Draft LRDP June 2005 RECOMMENDED for the UI Client to provide some UI hints to the user that the initialisation operation is a lengthy one, i.e. hourglass mouse cursor. 2.2 uiFragment Format The element contains the description of parts of the user interface in terms of widgets and styles. These kind of messages are created by applications that are not aware of the whole structure of the UI when the application is instantiated. As the application becomes aware of the structure of the UI, it MUST send the partial UI description to the Viewer as an message. All immediate child elements of the element MUST be namespace-qualified and MUST belong to the namespaces that were negotiated during the Session Setup step. 2.3 uiUpdate Format The element contains the change in the state of the UI Model describing the new state of a widget element properties. An message MUST follow an or message. All immediate child elements of the element MUST be namespace-qualified and MUST belong to the namespaces that were negotiated during the Session Setup step. 2.4 uiEvent Format The element contains information about an event that was triggered due to the user interaction with the Viewer. Currently, the has one child element that contains the name of the event. 3. Formal XML Syntax LRDP is specified in XML Schema notation. The formal syntax presented here is a complete schema representation of LRDP suitable for automated validation of LRDP XML instances. Stirbu Expires December 31, 2005 [Page 7] Internet-Draft LRDP June 2005 Lightweight Remote Display Protocol (LRDP) Schema v1 Stirbu Expires December 31, 2005 [Page 8] Internet-Draft LRDP June 2005 4. IANA Considerations This document uses a proposed XML namespace and schema. Accordingly, the following registration information is provided for the IANA: o URN/URI: * urn:ietf:params:xml:ns:lrdp1 o Contact: * Vlad Stirbu o XML: * The XML Schema specified in Section 3. 5. Security Considerations Input and output data may be highly sensitive. For example, input data may contain a user's personal data, such as passwords. At the same time, the user needs to be sure that the UI he wants to interact with was not tampered with. Other applications may not require confidentiality in their UI output. The LRDP XML layer provides no authentication or privacy facilities of its own. It relies on the transport layer for all of these abilities. 6. Acknowledgements 7. References 7.1 Normative References [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [2] World Wide Web Consortium, "Extensible Markup Language (XML) 1.0 (Third Edition)", W3C Recommendation XML, February 2004, . [3] World Wide Web Consortium, "XML Schema Part 1: Structures Second Edition", W3C Recommendation XML Schema, October 2004, < http://www.w3.org/TR/2004/REC-xmlschema-1-20041028/ >. [4] World Wide Web Consortium, "XML Schema Part 2: Datatypes Second Edition", W3C Recommendation XML Schema, October 2004, . [5] World Wide Web Consortium, "Namespaces in XML", W3C Recommendation XML Namespaces, January 1999, . Stirbu Expires December 31, 2005 [Page 9] Internet-Draft LRDP June 2005 [6] Rose, M., "The Blocks Extensible Exchange Protocol Core", RFC 3080, March 2001. 7.2 Informative References [7] Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., and M. Stal, "Pattern-Oriented Software Architecture, Volume 1: A System of Patterns", John Wiley & Sons , August 1996. [8] Handley, M. and V. Jacobson, "SDP: Session Description Protocol", RFC 2327, April 1998, . [9] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002, . [10] UPnP Forum, "Remote UI Client and Server V 1.0", August 2004, . [11] Abrams, M. and J. Helms, "User Interface Markup Language (UIML) Specification", Working Draft wd-UIML-UIMLspecification-3.1, OASIS , March 2004. [12] Urpalainen, J., "An Extensible Markup Language (XML) Patch Operations Framework Utilizing XPath Selectors", draft-urpalainen-simple-xml-patch-ops-00 (work in progress), March 2005. Author's Address Vlad Stirbu Nokia Visiokatu 3 Tampere 33720 Finland Phone: +358 7180 60572 Email: vlad.stirbu@nokia.com Appendix A. UI Server Consideration If an Application supports multiple UI Client platforms then the UI Model has two components: Generic UI Model and UI Model. Stirbu Expires December 31, 2005 [Page 10] Internet-Draft LRDP June 2005 +-------------------------------+ | UI Server | | | | +---------------------------+ | | | Application | | | | +-----------------------+ | | | | | Application Logic | | | | | +-----------------------+ | | | | +----------+ +----------+ | | | | | Generic | | UI Model | | | | | | UI Model | | | | | | | +----------+ +----------+ | | | +---------------------------+ | | +---------+ +------+ | | | Session | | LRDP | | | | Setup | | | | | +---------+ +------+ | | | +-------------------------------+ The Generic UI Model is described using an abstract meta-language, User Interface Markup Language (UIML) [11], that has two important characteristics: o Provides the canonical format for describing user interfaces that map to multiple devices. o Generate one description of the user interface connection to application logic independent of target device. The UI Model is described using the platform specific UI description language. The UI Model is obtained by converting the Generic UI Model using an UIML vocabulary specific to the UI Client platform. At present, the list of standard UIML vocabularies is posted on UIML website: . Appendix B. Examples In this examples, the user interfaces are described using an imaginary "UI Description Example" XML Schema. B.1 Initialisation The UI Server MUST send an message to the UI Client after the Session Setup has completed and the LRDP session is established. The message contains the full description of the UI. Stirbu Expires December 31, 2005 [Page 11] Internet-Draft LRDP June 2005 When the UI Client receives the message, it starts building the widget tree. During the rendering process the widgets are replaced with the equivalent platform representation of them and the UI may look like this: +-------------------------------------------------------------+ | Hello World! | +-------------------------------------------------------------+ | | | +------------------+ +----------++----------+ | | Change title: | New Title | | Update || Cancel | | | +------------------+ +----------++----------+ | | | +-------------------------------------------------------------+ B.2 Client Updates Consider that the user is staring to type the new name for the window inside the textbox area. Stirbu Expires December 31, 2005 [Page 12] Internet-Draft LRDP June 2005 +-------------------------------------------------------------+ | Hello World! | +-------------------------------------------------------------+ | | | +------------------+ +----------++----------+ | | Change title: | Updated Title_ | | Update || Cancel | | | +------------------+ +----------++----------+ | | | +-------------------------------------------------------------+ When the user has finished typing and textbox has lost the focus, the UI Client MUST inform the UI Server that the content of the "value" attribute has changed. The UI Client is doing so by sending an message to the UI Server indicating only the change. The content of the MAY be formated as an XML patch [12]. Update Title B.3 Client Event When the user is pressing the "Update" button, the Host Windowing System captures the event and passes it to the Viewer which notifies the UI Server with an message that the "UpdatePressed" event occurred on the rendering side. UpdatePressed The UI Server receives an event form the UI Client it passes it to the Application Logic for processing. The Application Logic takes the appropriate measures for treating the event and updates the UI Model with the new value for the window title. Stirbu Expires December 31, 2005 [Page 13] Internet-Draft LRDP June 2005 B.4 Server Update When the UI Model is updated the UI Server informs the UI client about the change by sending an message. The content of the MAY be formated as an XML patch [12]. Update Title The UI Client receives the message and updates the UI Model and renders the new UI with the new window title: +-------------------------------------------------------------+ | Updated Title | +-------------------------------------------------------------+ | | | +------------------+ +----------++----------+ | | Change title: | Updated Title | | Update || Cancel | | | +------------------+ +----------++----------+ | | | +-------------------------------------------------------------+ Stirbu Expires December 31, 2005 [Page 14] Internet-Draft LRDP June 2005 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Stirbu Expires December 31, 2005 [Page 15]